Lean proof validity
- Score
- 22.1
- Wallet spend
- $37.45 of $200
- Model cost
- $0.80
- Agent time
- 7h 19m
- Steps
- 394
- Tokens in / out
- 36.2M / 293k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 9th of 19
Score
22.1raw metric 0.6105
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads a causal LM with AutoModelForCausalLM.from_pretrained(MODEL_DIR) and runs a batched forward pass over th
server.py loads a causal LM with AutoModelForCausalLM.from_pretrained(MODEL_DIR) and runs a batched forward pass over the chat-templated prompt built in prompt.py. The verdict is valid = logit[Yes] > logit[No] at the last prompt position. No heuristic or rule-based path decides the result.
✓no lean execution or reimplementationThe serving path in serve.sh and server.py has no subprocess calls, Lean binary or Lean source. The trusted no_lean audi
The serving path in serve.sh and server.py has no subprocess calls, Lean binary or Lean source. The trusted no_lean audit passed with no violations and found 0 Lean source files. The development session used Lean only to label training data on a separate compute VM, and that Lean setup is not part of the runtime.
✓stateless per requestserver.py queues each (request, lean) pair, runs dynamic batching only for throughput, and returns {"valid": bool} from
server.py queues each (request, lean) pair, runs dynamic batching only for throughput, and returns {"valid": bool} from that item's own logits. There is no response cache, no cross-request statistics and no adaptive thresholding. Warmup is one dummy forward pass.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$DIR/model with HF_HUB_OFFLINE=1, and server.py loads from MODEL_DIR. The shipped model/*.safete
serve.sh sets MODEL_DIR=$DIR/model with HF_HUB_OFFLINE=1, and server.py loads from MODEL_DIR. The shipped model/*.safetensors inventories show a Qwen3 bf16 architecture (36 layers, hidden size 4096, q_norm/k_norm, vocab 151936). Trajectory steps 361-368 show the c1 LoRA adapter merged into Qwen3-8B (c1_merged), served and evaluated with the same server, uploaded and then downloaded into output/model/.
✓training data provenanceThe dsC corpus combines four kinds of data: the shipped validation.jsonl; synthetic mutants and LLM variants of its stat
The dsC corpus combines four kinds of data: the shipped validation.jsonl; synthetic mutants and LLM variants of its statements, labelled with the participant's own Lean (gen.py, make_llm_stmts.py, llm_prove.py); seeds_val/seeds_inv, which contain shipped validation statements; and the public Goedel-LM/Lean-workbook-proofs train parquet, downloaded with curl from Hugging Face (extract_lwp.py, run_data.sh, trajectory#34-58 and later steps). These sources are declared in architecture.json and evidence/data_manifest.json. The data manifest mislabels the seed files as Lean-workbook; that is a labelling slip, not an undeclared source. A 'minif2f' Hugging Face dataset search (trajectory#25) was never followed by a download or any use. No external miniF2F copies or attempts were found.
✓non specialist initializationarchitecture.json declares Qwen/Qwen3-8B at revision b968826d. The trajectory shows snapshot_download('Qwen/Qwen3-8B') i
architecture.json declares Qwen/Qwen3-8B at revision b968826d. The trajectory shows snapshot_download('Qwen/Qwen3-8B') into /root/models/Qwen3-8B. train_c.sh and train.py run --model /root/models/Qwen3-8B with a peft LoRA, and merge2.py merged that adapter into the same base (trajectory#361). This general base is allowed and is not a specialist validity model.
✓harness credentials used only by the harnessThe only LLM API use was through the permitted Spin OpenRouter gateway with SPIN_KEY: llm_prove.py, plus the zero-shot j
The only LLM API use was through the permitted Spin OpenRouter gateway with SPIN_KEY: llm_prove.py, plus the zero-shot judge test in trajectory#14. HF_TOKEN was used only for Hugging Face downloads, which is allowed. The ANTHROPIC_API_KEY appeared only in an environment listing with values hidden (trajectory#8) and was never used.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed7h 6m into the runtook 6m 58s
startup 7.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 133.8 s plus 0.1 s for the first request, 26.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed7h 13m into the runtook 6m 20s
startup 9.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 132.2 s plus 0.2 s for the first request, 26.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final summary of what was delivered:
Solution
/workspace/output/ contains a 16.4 GB submission (59 files): a Qwen3-8B model (revision b968826d9c46...) fine-tuned with LoRA as a Yes/No verbalizer over the whole Lean file, merged into bf16 safetensors, plus serve.sh, server.py, prompt.py, and full data/training evidence.
Results
| Measurement | Macro F1 | Notes |
|---|---|---|
| Provided validation attempts (526) | 0.9639 | in the shipped model's training data (allowed by the task) |
| 13-theorem holdout attempts (146) | 0.9726 | same |
| Unseen-theorem estimate (model B, 13 theorems removed) | 0.664 | honest generalization number |
| Self-check, 3,540 public requests at concurrency 64 | — | 132 s, 0 failed requests, 26.7 req/s (600 s allowed) |
Startup 9.2 s (300 s allowed); peak GPU memory ~57 GiB of 80 GiB.
How the labels were made
Every training label came from compiling the file with Lean v4.15.0 + Mathlib v4.15.0 during development (never at inference): 33,707 synthetic variants (renamed variables, changed numerals, 12 proof mutations) over seeds from the provided 50 theorems and the Lean-workbook-proofs train split, plus 385 LLM-generated attempts (deepseek-v4-pro/flash via Spin), class-balanced into a 37,692-row training file whose gzip and sha256 ship in evidence/corpora/. No miniF2F material was used beyond the provided validation file itself.
Two self-checks both passed; the final artifact was verified byte-identical to the model measured on the development GPU.