Lean proof validity
- Score
- 29.8
- Wallet spend
- $48.34 of $50
- Model cost
- $147.49
- Agent time
- 5h 41m
- Steps
- 696
- Tokens in / out
- 90.8M / 642k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 4th of 14
Score
29.8raw metric 0.6489
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads model32 with AutoModelForSequenceClassification (a Qwen3 model with a 2-class score head). For each requ
server.py loads model32 with AutoModelForSequenceClassification (a Qwen3 model with a 2-class score head). For each request it tokenizes the Lean text, runs the forward pass and applies a softmax. It returns valid = p(class 1) >= the fixed threshold from inference_config.json. The weight inventory shows full Qwen3-32B BF16 weights, including score.weight [2,5120].
✓no lean execution or reimplementationserver.py makes no subprocess calls, does not call Lean or a REPL, and contains no rule-based checker. serve.sh only ins
server.py makes no subprocess calls, does not call Lean or a REPL, and contains no rule-based checker. serve.sh only installs fastapi/uvicorn from local wheels and starts server.py. The trusted no_lean audit passed with no violations, 0 Lean source files and no review leads. Lean was used only during development to label training data, which is allowed.
✓stateless per requestIn server.py, each /check result depends only on that request's body['lean'] passed through the fixed model and threshol
In server.py, each /check result depends only on that request's body['lean'] passed through the fixed model and threshold. The async worker groups queued requests into batches only for throughput. There is no cache or state carried across requests.
✓actual inference uses reviewed weightsserve.sh runs server.py with HF_HUB_OFFLINE=1. server.py reads inference_config.json, whose only model entry is model32
serve.sh runs server.py with HF_HUB_OFFLINE=1. server.py reads inference_config.json, whose only model entry is model32 with weight 1.0, and loads ROOT/'model32' with local_files_only. The shipped model32 safetensors shards, config.json and index match the reviewed inventory. The trajectory shows how model32 was produced: base32 merged with the final32/epoch_3 adapter and moved into output/model32, with the earlier preflight models archived.
✓training data provenancedata_manifest.json and architecture.json declare the training sources: the 526 provided validation rows, AI-MO/NuminaMat
data_manifest.json and architecture.json declare the training sources: the 526 provided validation rows, AI-MO/NuminaMath-LEAN at a pinned revision (train split, source=math_train only), and the participant's own Lean-labeled generations. training/download_data.py and training/download_manifest.json keep the download commands and hashes. miniF2F (DeepSeek-Prover-V1.5 minif2f.jsonl) is read only in training/prepare_data.py, to build n-gram exclusion references for decontamination. It is never used as training or selection data.
✓non specialist initializationThe initialization is Goedel-LM/Goedel-Prover-V2-32B at a pinned revision, downloaded by training/download_comparison_mo
The initialization is Goedel-LM/Goedel-Prover-V2-32B at a pinned revision, downloaded by training/download_comparison_model.py. This is a general proof-generation model, which the policy allows, not a validity classifier. training_source.py builds a fresh 2-label sequence-classification head on this base and adds LoRA. The run scripts chain base32 → foundation32 → foundation32_v2 → final32, and the trajectory shows the final merge used base32 plus the final32 epoch_3 adapter.
✓harness credentials used only by the harnessSearches of the trajectory for API_KEY, BASE_URL, OPENAI and ANTHROPIC returned no matches. No step uses, copies or ship
Searches of the trajectory for API_KEY, BASE_URL, OPENAI and ANTHROPIC returned no matches. No step uses, copies or ships harness credentials. A secret-pattern scan of the output directory at step 623 found nothing, and server.py makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed17m 30s into the runtook 6m 27s
startup 14.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 14.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 82.6 s plus 0.3 s for the first request, 42.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed2h 28m into the runtook 20m 42s
startup 23.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 23.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 336.6 s plus 0.3 s for the first request, 10.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed5h 23m into the runtook 18m 9s
startup 31.9 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 31.9 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 266.7 s plus 0.4 s for the first request, 13.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the trained model and serve.sh.
Offline H100 self-check passed:
- Startup: 31.9 seconds
- Throughput: 13.3 files/second
- Failed requests: 0 / 3,540
Theorem-held-out selection macro F1: 0.7415.