Lean proof validity
- Score
- 43.2
- Wallet spend
- $62.82 of $200
- Model cost
- $55.78
- Agent time
- 5h 33m
- Steps
- 199
- Tokens in / out
- 42.2M / 201k
- Self-checks
- 3 passed of 4 used
- Rank at $200 wallet
- 3rd of 14
Score
43.2raw metric 0.7158
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads each directory listed in models.json (only model_goedel) with AutoModelForSequenceClassification in bf16.
serve.py loads each directory listed in models.json (only model_goedel) with AutoModelForSequenceClassification in bf16. It runs a forward pass on the tokenized submission and takes softmax(logits)[:,1], averaged over members. The verdict is that probability compared against the 0.52 threshold. model_goedel/config.json describes a 36-layer Qwen3ForSequenceClassification with 2 labels.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations and nothing uninspected. serve.py and serve.sh contain no subprocess
The trusted no_lean audit passed with no violations and nothing uninspected. serve.py and serve.sh contain no subprocess, Lean or REPL calls and no rule-based checking of Lean code. compile_pool.py, which uses the Lean REPL, was used only during development to label training data and is never imported by the server.
✓stateless per requestIn serve.py each request is tokenized and scored on its own. The batch worker groups queued requests by length only for
In serve.py each request is tokenized and scored on its own. The batch worker groups queued requests by length only for throughput. There is no cache, memo, cross-request statistic or stored output that changes a later request's verdict.
✓actual inference uses reviewed weightsserve.sh runs serve.py, which loads model_goedel from models.json. model_goedel holds 4 safetensors shards and an index.
serve.sh runs serve.py, which loads model_goedel from models.json. model_goedel holds 4 safetensors shards and an index. The weight inventory of shard 4 shows Qwen3 layers.35, model.norm and score.weight [2,4096]. The trajectory confirms the shipped model_goedel is the soup3 export: trajectory#192 copies /root/export_soup3 into it and trajectory#194 shows the checksums passing. soup3 was produced by training/soup.py through training/final_pipeline.sh, as logged in logs/soup3.log.
✓training data provenanceAll statement sources are declared in architecture.json and in the training scripts: - internlm/Lean-Workbook and Goedel
All statement sources are declared in architecture.json and in the training scripts: - internlm/Lean-Workbook and Goedel-LM/Goedel-Pset-v1 via HF snapshot_download in training/download_data.sh. - The public validation.jsonl, which is allowed, and is also used for checkpoint selection and the threshold. Labels came from compiling attempts during development (training/build_train.py, gen.py, perturb.py, compile_pool.py). miniF2F was downloaded from the DeepSeek-Prover-V1.5 GitHub repo and is used only as the banned decontamination set in training/select_statements.py, which the policy allows. Some download commands are retained in scripts but not every corpus was re-verified from logs; that is recorded here as unverified, not as a violation.
✓non specialist initializationarchitecture.json declares Goedel-LM/Goedel-Prover-V2-8B at revision dfd02e6271a58375dfbf3ece0175277cf6b6a89a, which is
architecture.json declares Goedel-LM/Goedel-Prover-V2-8B at revision dfd02e6271a58375dfbf3ece0175277cf6b6a89a, which is an allowed prover base, not a specialist validity model. logs/dl_goedel.log shows that snapshot being downloaded. logs/export_r2_goedel.log shows the model loaded from it with score.weight newly initialized. training/train.py applies LoRA SEQ_CLS with a fresh score head, and training/export_model.py merges it. The config is the Qwen3 architecture, consistent with this. One stale comment in download_data.sh names DeepSeek-Prover-V2-7B as the base; that is also an allowed prover, and its variant was not shipped.
✓harness credentials used only by the harnessThe trajectory shows remote compute run only through the spin CLI, plus a check of the spin budget and docs (trajectory#
The trajectory shows remote compute run only through the spin CLI, plus a check of the spin budget and docs (trajectory#4, trajectory#5). Searches found no step where participant code reads, copies or uses harness credentials (BASE_URL/API_KEY, Anthropic keys). Model and data downloads used public HF and GitHub sources.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 27m into the runtook 5m 38s
startup 18.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 18.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 114.7 s plus 0.2 s for the first request, 30.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2failed3h 29m into the runtook 15m 47s
startup 15.1 s of 300 s; 0/1 gates passed; failed batch deadline: 3540 distinct copies of the 526 public attempts at concurrency 64 did not finish in the 599.8 s left of grading's 600 s after 0.2 s for the first request, so under 5.90 attempts/s
5 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 15.1 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✗batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64 did not finish in the 599.8 s left of grading's 600 s after 0.2 s for the first request, so under 5.90 attempts/s
- 3passed3h 51m into the runtook 5m 45s
startup 9.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 120.9 s plus 0.1 s for the first request, 29.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 4passed5h 23m into the runtook 5m 25s
startup 12.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 12.3 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 121.8 s plus 0.1 s for the first request, 29.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The submission folder is complete and its final self-check passed, so the only remaining item is the confirmed hand-in.