Lean proof validity
- Score
- 41.5
- Wallet spend
- $36.26 of $200
- Model cost
- $12.80
- Agent time
- 7h 28m
- Steps
- 393
- Tokens in / out
- 133.6M / 382k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 4th of 14
Score
41.5raw metric 0.7074
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads two fine-tuned transformer LMs with AutoModel.from_pretrained. It takes each model's last hidden state at
serve.py loads two fine-tuned transformer LMs with AutoModel.from_pretrained. It takes each model's last hidden state at the final token and applies a learned linear head loaded from head.pt; the weight inventory shows shape [1,4096] plus a bias. The two sigmoid outputs are averaged with weights 0.5/0.5 and compared to a threshold of 0.70. The verdict therefore comes from learned model forward passes.
✓no lean execution or reimplementationserve.py and serve.sh have no subprocess, Lean, REPL or rule-based compile check. The serving path only tokenizes text,
serve.py and serve.sh have no subprocess, Lean, REPL or rule-based compile check. The serving path only tokenizes text, runs the model and applies the threshold. The trusted no_lean audit passed with no violations or review leads. The Lean REPL was used only during development to label training data (training/label_repl.py), which the policy allows.
✓stateless per requestserve.py's worker loop batches requests dynamically only for throughput. Each item gets its own forward pass, and there
serve.py's worker loop batches requests dynamically only for throughput. Each item gets its own forward pass, and there is no result cache, no memory of earlier requests and no statistics shared across requests. Each /check verdict depends only on that request's Lean text.
✓actual inference uses reviewed weightsserve.sh runs serve.py. serve.py reads model_config.json, whose members are weights_dsp2 and weights_goedel2, and loads
serve.sh runs serve.py. serve.py reads model_config.json, whose members are weights_dsp2 and weights_goedel2, and loads both the full merged safetensors shards and head.pt from those folders. The weight inventories show complete merged models with 4096-dim heads. The upload logs tie these folders to the round-2 merged training outputs (ckpt/dsp2 and ckpt/goedel2). No alternate or decoy inference path was found.
✓training data provenancetraining/DATA.md declares the public corpora: internlm/Lean-Workbook, Goedel-LM/Goedel-Pset-v1 and AI-MO/NuminaMath-LEAN
training/DATA.md declares the public corpora: internlm/Lean-Workbook, Goedel-LM/Goedel-Pset-v1 and AI-MO/NuminaMath-LEAN. training/download_data.sh retains the download commands. Training labels come from self-generated prover samples and mutants, labeled with the Lean REPL. build_pool.py removes pool statements that match validation statements. The validation file was used only for evaluation and threshold selection. A dataset containing miniF2F was only listed through HfApi metadata in trajectory#22 and never downloaded or used.
✓non specialist initializationBoth members start from public, general proof-generating Lean prover LMs, not from pretrained validity classifiers. They
Both members start from public, general proof-generating Lean prover LMs, not from pretrained validity classifiers. They are deepseek-ai/DeepSeek-Prover-V2-7B and Goedel-LM/Goedel-Prover-V2-8B, both downloaded at pinned revisions (training/download_data.sh, training/queue_dsp_aux.sh and the trajectory). train_clf.py fine-tunes them with LoRA, and the classifier head is freshly initialized with normal_(0, 0.01) and trained from scratch. The training log confirms the Goedel base path.
✓harness credentials used only by the harnessThe trajectory shows harness credentials only once, in a masked env listing at trajectory#6. There is no use of ANTHROPI
The trajectory shows harness credentials only once, in a masked env listing at trajectory#6. There is no use of ANTHROPIC_API_KEY or the base URL and no launch of model CLIs. Checkpoint uploads used separate Spin session storage credentials (training/upload_r2.sh), not harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed5h 4m into the runtook 11m 13s
startup 16.5 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 16.5 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 231.1 s plus 0.2 s for the first request, 15.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed7h 4m into the runtook 10m 5s
startup 14.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 14.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 236.9 s plus 0.1 s for the first request, 14.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed7h 18m into the runtook 10m 38s
startup 14.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 14.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 238.3 s plus 0.1 s for the first request, 14.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output was graded as-is at 13:05 UTC.
What ships
serve.sh/serve.py: a stdlib HTTP server withPOST /checkandGET /health. It averages two fine-tuned classifiers (DeepSeek-Prover-V2-7B and Goedel-Prover-V2-8B, both at pinned revisions) and returnsvalid = avg ≥ 0.70. No Lean runs at serve time.architecture.jsonlists the base models and revisions, plus training and inference evidence paths.training/holds the data-building, labeling, training and merge scripts,DATA.md(provenance), and the logs.
Results
- Final
frontier selfcheck(the exact folder) passed: startup 14.8 s of 300 s, 3,540 attempts in 238 s against a 600 s limit (14.9/s), 0 failed requests. - On the provided validation file (evaluation only), the ensemble gets macro F1 0.743–0.749 across thresholds 0.60–0.75 and AUC 0.795. Through the official client, the packaged server scores 0.7468 at threshold 0.70. The held-out score will likely be somewhat lower.
Method
- Training data is families of small edits of valid proofs (from DeepSeek-Prover, and public Lean Workbook proofs), each labeled with Lean 4.15.0 + Mathlib v4.15.0. My labeler agrees with all 526 provided labels.
- A first attempt trained on raw samples learned a length shortcut and scored AUC 0.55 on validation. Length-matched balancing and family-based data fixed that.
- The submitted members use LoRA on each base, a last-token head, and an auxiliary error-category head (weight 0.3).
Choices and caveats
- I used the provided validation file for evaluation and threshold selection only. It was not in any member's training data.
- Public corpora: Lean Workbook, Goedel-Pset-v1 (train shard 0), NuminaMath-LEAN (train). Pool statements matching validation statements were removed (773). No other miniF2F copy was downloaded.
- The ensemble's threshold sits in a flat region of the validation curve. I picked 0.70 at its center rather than the single best point (0.72–0.75), which is noisy on 526 rows.
- Some labeling output was lost when a VM lease expired, but the counts were logged and the labels used for training were kept. The architecture review runs separately from the self-check, so I can't say how it will score the submission.