Lean proof validity
- Score
- 41.8
- Wallet spend
- $8.41 of $10
- Model cost
- $30.63
- Agent time
- 9h 29m
- Steps
- 660
- Tokens in / out
- 68.5M / 495k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 1st of 14
Score
41.8raw metric 0.7092
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passThe verdict comes from a learned model. server.py loads a LlamaModel (DeepSeek-Prover-V2-7B with a merged LoRA) from mod
The verdict comes from a learned model. server.py loads a LlamaModel (DeepSeek-Prover-V2-7B with a merged LoRA) from model/ and runs a forward pass on the featurized text. It takes the last-token hidden state, applies the linear head from head.safetensors ([1,4096] weight plus bias) and returns valid when logit > threshold, which is 0.1 in classifier.json. featurize.py only strips the header and comments and wraps the text in a prompt; it contains no rules that decide validity.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations and no review leads. server.py and serve.sh start no subprocess and
The trusted no_lean audit passed with no violations and no review leads. server.py and serve.sh start no subprocess and make no Lean or REPL call, and featurize.py does no proof checking. Lean v4.15.0 REPL compiles were used only during development to create training labels.
✓stateless per requestEach request gets its own score. server.py batches requests dynamically for throughput only, and each request's logit is
Each request gets its own score. server.py batches requests dynamically for throughput only, and each request's logit is computed independently from its own text. There is no cache, memo, cross-request statistic or adaptive threshold; the threshold is a constant loaded once at startup.
✓actual inference uses reviewed weightsThe serving path loads the reviewed weights. serve.sh runs server.py with --model_dir pointing at the submission's model
The serving path loads the reviewed weights. serve.sh runs server.py with --model_dir pointing at the submission's model/ directory. server.py calls LlamaModel.from_pretrained on that directory, which loads model/model.safetensors (12.98 GB of BF16 LlamaModel tensors in the inventory), and it loads model/head.safetensors. pack_model.py copies these files from the merged soup output produced by soup.py, which matches the inventoried weights.
✓training data provenanceThe sources are declared. dl_data.py and DATA_MANIFEST.md name the public Goedel-LM/Lean-workbook-proofs train split and
The sources are declared. dl_data.py and DATA_MANIFEST.md name the public Goedel-LM/Lean-workbook-proofs train split and the Goedel-LM/Goedel-Pset-v1 train shard, and prep_remote.py excludes miniF2F-style statement names. The other training data comes from three places: proof attempts generated by the declared prover models, mutated and renamed variants made by mutate.py, mk_gen3.py and mk_gen5.py, and the supplied public validation file (mk_val.py). That validation file has a declared split: 20 held-out theorems are used only for selection and the threshold. The labels are Lean compile results from development. The trajectory shows downloads from these public HF sources only (internlm/Lean-Workbook was also downloaded but only inspected). No forbidden benchmark source was found.
✓non specialist initializationThe model starts from a general prover, not a validity classifier. train.py and soup.py load deepseek-ai/DeepSeek-Prover
The model starts from a general prover, not a validity classifier. train.py and soup.py load deepseek-ai/DeepSeek-Prover-V2-7B at the recorded revision (dl_models.py and dosoup.sh) with AutoModel.from_pretrained, apply LoRA and train a new head. That head is initialised from the base lm_head's ' Yes' minus ' No' rows. model/config.json matches the base architecture. The base is a general proof-generation model, not a pre-trained validity classifier.
✓harness credentials used only by the harnessOnly the Hugging Face token was used, as allowed. The trajectory shows HF_TOKEN, which the task prompt describes as a re
Only the Hugging Face token was used, as allowed. The trajectory shows HF_TOKEN, which the task prompt describes as a read-only download token, used only for HF metadata queries and model/dataset downloads; hf_env.sh was removed from the package. No step uses the harness's agent API keys (ANTHROPIC/OPENAI/base URLs) or launches model workers with them. Step 411 only ran a help command.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed3h 30m into the runtook 1m 40sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed3h 32m into the runtook 5m 1s
startup 11.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 11.1 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 82.1 s plus 0.2 s for the first request, 43.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed9h 24m into the runtook 4m 22s
startup 11.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 11.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 81.4 s plus 0.1 s for the first request, 43.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and accepted (07:58:53 UTC), so the session is over. The last self-check, on an H100, passed: start-up 11 s, 43 attempts/s, 0 failed requests. It reports no score, so the F1 below comes from my own validation runs.
- Model: DeepSeek-Prover-V2-7B with a LoRA fine-tune and a linear head, averaged over the last five snapshots of the final training phase. A file counts as valid when the logit is above 0.10.
- Held-out check: on the 20 provided-validation theorems I held out from training (224 files), AUC is 0.788 and macro-F1 is about 0.72. That set is small, so expect roughly ±0.03 on F1.
- Final training phase: it re-weights generator families so that proof style alone doesn't predict the label. The Qwen files in particular were 99.6% invalid and would have taught that shortcut.
- In
/workspace/output/:serve.sh,server.py,featurize.py,model/,train/,data_pipeline/,logs/,data/,DATA_MANIFEST.md,README.mdandarchitecture.json. There are no symlinks and no secrets. - Architecture review: it did not run in the self-check and only happens at grading. I made sure every file named in
architecture.jsonexists.