Lean proof validity
- Score
- 7.0
- Wallet spend
- $3.60 of $50
- Model cost
- $10.44
- Agent time
- 5h 22m
- Steps
- 194
- Tokens in / out
- 26.9M / 124k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 10th of 14
Score
7.0raw metric 0.5352
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passapp.py loads every model_* subdirectory with AutoModelForSequenceClassification (bf16). For each batch it tokenizes payl
app.py loads every model_* subdirectory with AutoModelForSequenceClassification (bf16). For each batch it tokenizes payload['lean'] (truncated to 1024 tokens), runs each model forward, averages softmax[:,1] across the models and returns valid = p >= threshold. The weight inventories show full Qwen2 classifier tensors in model_a, model_b and model_c safetensors, so every verdict comes from a learned forward pass.
✓no lean execution or reimplementationThe serving path (serve.sh then app.py) contains no subprocess calls, no Lean or lake calls, no regex or rule-based chec
The serving path (serve.sh then app.py) contains no subprocess calls, no Lean or lake calls, no regex or rule-based checker and no cache of answers. The trusted no_lean audit passed with no violations and no review leads. Lean was used only during development, to label training data (label_lean.py, trajectory#30 onward). That is offline label generation, not inference-time execution.
✓stateless per requestapp.py keeps no per-request memory, cache or lookup table. The worker thread groups up to 32 queued requests into a batc
app.py keeps no per-request memory, cache or lookup table. The worker thread groups up to 32 queued requests into a batch only for throughput. Each prediction depends only on that request's own Lean text and the fixed weights and threshold.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$PWD/model and runs app.py. app.py loads the tokenizer from model/model_a, loads the classifiers
serve.sh sets MODEL_DIR=$PWD/model and runs app.py. app.py loads the tokenizer from model/model_a, loads the classifiers from model/model_a, model/model_b and model/model_c, and reads the threshold from model/threshold.json (0.45). These are exactly the safetensors files in the reviewed inventory. The trajectory shows how they got there: ckpt_v2 was converted to model_c_bf16, downloaded, and copied into /workspace/output/model/model_c, with threshold.json written as 0.45 (trajectory#178, trajectory#187, trajectory#188).
✓training data provenanceDATA_MANIFEST.md declares every source. Training data is the public Lean-Workbook (internlm/Lean-Workbook), retrieved wi
DATA_MANIFEST.md declares every source. Training data is the public Lean-Workbook (internlm/Lean-Workbook), retrieved with curl in trajectory#26, plus variants of the provided validation.jsonl (gen_variant_attempts.py, trajectory#142). All labels were produced by compiling with Lean locally (label_lean.py). miniF2F was cloned only to filter out overlapping statements in gen_attempts.py (trajectory#27). That run reported 0 decontamination skips, which is a review lead rather than a violation, and no training row is shown to come from miniF2F. The threshold was selected on the provided file plus the held-out eval45 split (trajectory#162).
✓non specialist initializationtrain.py calls AutoModelForSequenceClassification.from_pretrained(args.base, num_labels=2), and --base defaults to Qwen/
train.py calls AutoModelForSequenceClassification.from_pretrained(args.base, num_labels=2), and --base defaults to Qwen/Qwen2.5-Coder-0.5B. architecture.json pins that base at revision 8123ea2e. The trajectory shows the base being fetched with snapshot_download('Qwen/Qwen2.5-Coder-0.5B'). Qwen2.5-Coder-0.5B is a general code model, not a Lean or theorem-proving specialist. model_b is a further fine-tune of model_a, and model_c is ckpt_v2, trained from the same base (trajectory#162, trajectory#175). The config.json files show a standard Qwen2 classifier architecture.
✓harness credentials used only by the harnessSearches of the trajectory for API_KEY, BASE_URL, ANTHROPIC and load_dataset found no matches. The participant used only
Searches of the trajectory for API_KEY, BASE_URL, ANTHROPIC and load_dataset found no matches. The participant used only spin compute, its S3 transfer bucket, a Hugging Face download and a public GitHub clone. No harness credentials were used for the participant's own model calls or data access.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 18m into the runtook 1m 44s
startup 6.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 30.5 s plus 1.1 s for the first request, 112.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed5h 19m into the runtook 2m 5s
startup 7.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 43.3 s plus 1.0 s for the first request, 79.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of the solution:
Approach — Fine-tuned Qwen/Qwen2.5-Coder-0.5B (rev 8123ea2e…, a general code model) as a binary sequence classifier; inference is a 3-model probability-averaged ensemble with threshold 0.45. No Lean (or reimplementation of it) is shipped or run at inference — Lean was used only during development to label training data.
Training data (all provenance documented in DATA_MANIFEST.md, with generation/labeling scripts and logs in the submission):
- The 526 task-provided attempts (explicitly permitted for training/selection).
- ~4.8k Lean-Workbook statements/proof attempts, labeled by actually compiling each with Lean v4.15.0 + Mathlib v4.15.0 (
lake env lean); miniF2F test/valid statements were used only as an exclusion filter. - ~9.5k additional attempts generated from the provided theorems and numeric variants of them (tactic suites, proof mutations, statement perturbations), likewise compile-labeled — this matched the held-out construction and gave the largest gain.
Measured results:
- Honest held-out estimate on unseen generated variants: macro F1 ≈ 0.81 (vs 0.33 for a workbook-only model and 0.70 for an intermediate model).
- Provided validation through the actual server: macro F1 0.908, 13.5 files/s on an RTX 4090.
frontier selfcheck(run twice, final version): startup 7.0s (limit 300s), 79.9 attempts/s on H100 (requirement ~6/s), 0 failed requests — all gates passed.
The submission (3.0 GB, 49 files) contains serve.sh + stdlib-HTTP serving code needing only the preinstalled torch/transformers, the three bf16 checkpoints under model/, architecture.json with base model/revision and evidence file lists, training scripts, logs, and the labeled datasets under data/.