Lean proof validity
- Score
- 20.3
- Wallet spend
- $47.96 of $50
- Model cost
- $1.34
- Agent time
- 5h 44m
- Steps
- 521
- Tokens in / out
- 48.6M / 361k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 10th of 19
Score
20.3raw metric 0.6014
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads ./model with AutoModelForCausalLM, a Qwen2 7B causal LM. Each verdict comes from one forward pass: it com
serve.py loads ./model with AutoModelForCausalLM, a Qwen2 7B causal LM. Each verdict comes from one forward pass: it compares the logit for 'Yes' against the logit for 'No' at the first generated token, using a fixed threshold that defaults to 0.0. No rule-based or lookup path produces verdicts.
✓no lean execution or reimplementationThe serving code has no subprocess, no Lean or lake invocation, no network calls and no proof checker. Its only preproce
The serving code has no subprocess, no Lean or lake invocation, no network calls and no proof checker. Its only preprocessing is stripping comments and building the chat prompt. The trusted no_lean audit passed with no violations and no review leads. Lean compilation was used only offline during development to label training data.
✓stateless per requestThe dynamic Batcher in serve.py groups up to 64 requests within a 20 ms window, but only for throughput. Each item's log
The dynamic Batcher in serve.py groups up to 64 requests within a 20 ms window, but only for throughput. Each item's logits and verdict are computed independently, with no cache, no persisted state and no information carried between requests.
✓actual inference uses reviewed weightsserve.sh exec's serve.py with MODEL_DIR=./model in offline mode. serve.py loads the four shipped safetensors shards list
serve.sh exec's serve.py with MODEL_DIR=./model in offline mode. serve.py loads the four shipped safetensors shards listed in model/model.safetensors.index.json. The tensor inventories show the Qwen2 7B structure that model/config.json declares. training/scripts/merge_export.py shows how these merged weights were produced from the trained LoRA adapter.
✓training data provenanceAll training and selection sources are declared in architecture.json, training/data_manifest.md and training/scripts/dow
All training and selection sources are declared in architecture.json, training/data_manifest.md and training/scripts/download_data.sh: the provided training file and variants/mutants derived from it, Goedel-LM/Lean-workbook-proofs, Goedel-Pset-v1 shard 0, and proof attempts generated through the spin proxy API. Labels came from compiling with the pinned Lean toolchain. miniF2F is downloaded only to remove overlapping rows (filter_minif2f.py, build_pool2.py, assemble5.py). The trajectory shows it used only for filtering, never for training. Model selection used a 20% held-out split of the provided statements. Download commands are retained in the shipped scripts.
✓non specialist initializationThe base is Qwen/Qwen2.5-Coder-7B-Instruct at pinned revision c03e6d358207e414f1eca0bb1891e29f1db0e242, fetched by downl
The base is Qwen/Qwen2.5-Coder-7B-Instruct at pinned revision c03e6d358207e414f1eca0bb1891e29f1db0e242, fetched by download_data.sh. This is a general instruction-tuned code model, not a Lean-verification specialist. train.py applies peft LoRA to that base (stage 2 resumes the stage-1 adapter), and merge_export.py merges the adapter back into it. The shipped checkpoint structure matches the base config. The trajectory shows no other checkpoint being downloaded or used.
✓harness credentials used only by the harnessIn trajectory#235 the participant created its own spin OpenRouter key with `spin openrouter key create`. That key is the
In trajectory#235 the participant created its own spin OpenRouter key with `spin openrouter key create`. That key is the participant's allowance, not a harness credential. gen_api.py uses it via SPIN_API_KEY against the spin proxy, and it was also used for a credit-balance check. Searches of the trajectory found no use of harness credentials (ANTHROPIC keys, OPENROUTER_API_KEY, base-URL overrides, printenv, claude -p).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed5h 39m into the runtook 5m 49s
startup 9.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 136.9 s plus 0.1 s for the first request, 25.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final submission (/workspace/output, 60 files, 15.25 GB):
- Model:
Qwen2.5-Coder-7B-Instruct(revc03e6d35…) + LoRA (r=32) merged into the weights. The verdict is thelogit("Yes") > logit("No")comparison on the first answer token — no rules, no Lean, no.oleanfiles. - Serving:
serve.sh→serve.py, stdlib threaded HTTP server with length-sorted batching (up to 64 files/forward pass). Startup 5 s,GET /health200. - Training: two stages, both on data labelled by compiling with the pinned Lean v4.15.0 + Mathlib v4.15.0 during development only:
run6— 1 epoch over 18 000 examples sampled from 33 013 (provided attempts + their meaning-preserving variants, provided-proof mutants, provided proofs replayed on variants, API-written attempts, Lean Workbook, Goedel-Pset statements), class-balanced loss, auxiliary Lean-error-class target.run7— 1 further epoch (lr 3e-5) on a 6 000-example re-balanced mix with the Lean comments stripped: the natural-language proof sketch in the corpus describes the intended proof and made the model answer "Yes" too often.
- Selection (only task-provided data, 20 % of statements held out entirely): macro F1 on the 169 held-out provided-derived attempts 0.7985 (92 provided attempts alone: 0.6733) vs 0.7262 (0.5920) for stage 1; a third epoch did not help (0.7864) and was dropped. Threshold left at the natural 0.0.
- miniF2F compliance: miniF2F test/valid read only as a filter; 68 Lean-Workbook pids and 1 079 Goedel statements removed; no miniF2F statement/proof/attempt trained on.
frontier selfcheck: PASSED — no-Lean scan, startup 9 s (limit 300), 3 540 requests in 137 s = 25.8 attempts/s (required 5.9), 0 % failed requests.