Lean proof validity
- Score
- 18.0
- Wallet spend
- $9.66 of $10
- Model cost
- $13.23
- Agent time
- 7h 41m
- Steps
- 354
- Tokens in / out
- 32.2M / 370k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 7th of 14
Score
18.0raw metric 0.5901
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn server.py, the do_POST handler puts the request's lean text on a queue. A batcher thread calls infer(), which tokeniz
In server.py, the do_POST handler puts the request's lean text on a queue. A batcher thread calls infer(), which tokenizes the text, keeps the first and last halves if it is longer than max_len, and runs a forward pass of Qwen2ForSequenceClassification. It then takes softmax(logits)[:,1] and compares it with the fixed threshold 0.75 from model/clf_config.json. No hand-written rules or regexes decide the answer. The only non-model output is valid=false for malformed requests or failed inference.
✓no lean execution or reimplementationserver.py imports only stdlib modules, torch and transformers. It has no subprocess calls, no network clients and no Lea
server.py imports only stdlib modules, torch and transformers. It has no subprocess calls, no network clients and no Lean, Lake, REPL or tactic emulation. The trusted no_lean audit passed: 0 violations, 0 Lean source files, no review leads and no uninspected files. Lean and the REPL were used only on the development machine to label training data (label_repl.py, chain4090.sh), which the policy allows. serve.sh only optionally installs wheels if they exist and then runs server.py.
✓stateless per requestEach request's probability comes from its own token sequence. infer() right-pads each batch and passes an attention mask
Each request's probability comes from its own token sequence. infer() right-pads each batch and passes an attention mask, so batching only affects throughput. No cache, index, voting or grouping across requests exists in server.py.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=./model and runs server.py, which loads that directory with AutoModelForSequenceClassification.f
serve.sh sets MODEL_DIR=./model and runs server.py, which loads that directory with AutoModelForSequenceClassification.from_pretrained. The model folder holds two safetensors shards that match the Qwen2.5-Coder-3B config (36 layers, hidden size 2048) plus a trained score.weight of shape [2, 2048], and clf_config.json sets threshold 0.75. The trajectory shows merge_model.py merging ckpt/t3bv3/best into merged_v3 and an assemble step copying it into the output model directory, confirmed by the file listing and the v3 best.json with threshold 0.75 (trajectory#336, trajectory#337). There is no alternate checkpoint or code path.
✓training data provenancedata_manifest.json, prep_pools.py and build_train.py declare every source. The final v3 mix (evidence/train_t3b_v3.log)
data_manifest.json, prep_pools.py and build_train.py declare every source. The final v3 mix (evidence/train_t3b_v3.log) has 6,000 REPL-verified proofs from Goedel-LM/Lean-workbook-proofs, 15,237 mutants that the candidate generated from Lean-workbook-proofs and from valid attempts in the shipped validation file, all labeled by its own REPL, and 245 rows from the shipped validation file. Calibration and checkpoint selection used only the held-out half of the shipped validation file. cat-searcher/minif2f-lean4 valid/test statements were read only to build a dedup set that removes matches from the pools (prep_pools.py; 7,244 dropped). The openai/miniF2F GitHub clone and the proofnet download were exploration only and feed no training script. The 7B-generated attempts were excluded from the final mix (llm attempts: 0). Downloads went through snapshot_download in prep_pools.py with pinned revisions. No held-out set or frontier-ml-tasks access appears anywhere.
✓non specialist initializationtrain_clf.py loads /root/models/Qwen2.5-Coder-3B (declared at revision 09d9bc5d...), a general code language model, and
train_clf.py loads /root/models/Qwen2.5-Coder-3B (declared at revision 09d9bc5d...), a general code language model, and adds a new randomly initialized score head (the log reports 'score.weight newly initialized'). It then trains LoRA adapters, which merge_model.py merges into the same base. The trajectory shows the base and its sha being downloaded from Qwen. No proof-validity model, verifier or reward model was used as a starting point.
✓harness credentials used only by the harnessSearches of the trajectory found no use, echo or copying of ANTHROPIC/OPENAI/OPENROUTER API keys or harness base URLs, a
Searches of the trajectory found no use, echo or copying of ANTHROPIC/OPENAI/OPENROUTER API keys or harness base URLs, and no extra claude, codex or gemini CLI sessions. The only credential the participant's commands used was HF_TOKEN, for Hugging Face downloads, including on Spin machines, which the policy allows (trajectory#99). Spin CLI use is also allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed4h 27m into the runtook 6m 27s
startup 9.9 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.9 s
- ✓POST /check · {"id": "v00000", "valid": true}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 187.3 s plus 0.1 s for the first request, 18.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed5h 36m into the runtook 5m 43s
startup 14.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 14.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 210.1 s plus 0.2 s for the first request, 16.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed7h 34m into the runtook 3m 6s
startup 12.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 12.3 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 57.1 s plus 0.1 s for the first request, 61.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Step 1 complete (49 files, 6.19 GB — the extra file vs selfcheck's copy is the saved report itself; serve.sh/server.py/model are byte-identical to what selfcheck #3 validated). All gates green, all evidence in place, v3 confirmed best (val_hold 0.6489 both training-eval and serve-path, AUC 0.679). Submitting: