Lean proof validity
- Score
- 3.0
- Wallet spend
- $35.35 of $200
- Model cost
- $25.71
- Agent time
- 9h 35m
- Steps
- 387
- Tokens in / out
- 36.0M / 248k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 10th of 14
Score
3.0raw metric 0.5149
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads AutoModelForCausalLM from MODEL_DIR (./model) in bf16 on CUDA. For each request it runs the transformer,
serve.py loads AutoModelForCausalLM from MODEL_DIR (./model) in bf16 on CUDA. For each request it runs the transformer, takes the last-position hidden state and applies lm_head, then returns valid = logit(yes) - logit(no) > THRESHOLD (0.0, set in serve.sh). So the decision comes from a learned model's forward pass. model/model.safetensors holds a full Qwen2 1.5B tensor set (embed 151936x1536, 28 layers).
✓no lean execution or reimplementationserve.py makes no subprocess, Lean or network calls. Its only logic is prompt formatting, truncation and the model forwa
serve.py makes no subprocess, Lean or network calls. Its only logic is prompt formatting, truncation and the model forward pass. The trusted no_lean audit passed with no violations or leads across 33 scanned files. Lean 4.15 + Mathlib was used only during development to label training data (training_code, DATA_MANIFEST.md), not in serving.
✓stateless per requestIn serve.py, each item's result in the throughput Batcher depends only on that item's own prompt and the fixed model wei
In serve.py, each item's result in the throughput Batcher depends only on that item's own prompt and the fixed model weights. There is no cache, memo or cross-request state that could affect predictions. The POST /check handler returns only {"valid": bool}.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$PWD/model and execs serve.py, which loads that directory. model/config.json (Qwen2ForCausalLM,
serve.sh sets MODEL_DIR=$PWD/model and execs serve.py, which loads that directory. model/config.json (Qwen2ForCausalLM, hidden 1536, 28 layers) and the model/model.safetensors inventory match the 1.5B checkpoint. The trajectory shows the packaging script wiping output/model and downloading /root/ckpt_15b into it (trajectory#77, trajectory#78). Self-check logs show the server 'loading model from /submission/model' (trajectory#79). One minor inconsistency: logs/train_final.log is actually the 7B run's log prefix, not the 1.5B log. That doesn't change the serving path or the shipped weights.
✓training data provenanceThe training data sources are declared in DATA_MANIFEST.md, architecture.json and training_code/prepare_statements.py: p
The training data sources are declared in DATA_MANIFEST.md, architecture.json and training_code/prepare_statements.py: public HF Lean Workbook / lean_workbook_proofs / Goedel-Pset train corpora, attempts generated with Goedel-Prover-SFT and DeepSeek-Prover-V1.5-RL and labeled with Lean, plus gold statements and mutations derived from the provided 526-row validation.jsonl (trajectory#211). A miniF2F jsonl was downloaded, but prepare_statements only keeps split=='train'. That file has only valid/test rows (trajectory#67), and the run printed 'minif2f_train statements: 0' (trajectory#66), so no miniF2F valid/test material entered training. Download commands are declared in the manifest. Where no retained download log exists, provenance is recorded as declared but unverified.
✓non specialist initializationtraining_code/train.py and training_code/final_chain.sh fine-tune Qwen/Qwen2.5-Coder-1.5B, a general code LM declared in
training_code/train.py and training_code/final_chain.sh fine-tune Qwen/Qwen2.5-Coder-1.5B, a general code LM declared in architecture.json at a pinned revision. The shipped checkpoint's structure (Qwen2, 1536 hidden, tied embeddings, 151936 vocab) matches that base. logs/selection.txt records choosing the 1.5B over the 7B and the ensemble. No specialist Lean-validity classifier was used as initialization. The prover models were used only to generate training attempts.
✓harness credentials used only by the harnessI searched the trajectory for credential use. The participant never read, exported or used harness API keys and never la
I searched the trajectory for credential use. The participant never read, exported or used harness API keys and never launched an agent or LLM client with them. The only appearance is a `ps aux` listing (trajectory#80, step 373) that incidentally showed the harness's own router command line with environment-variable names. Observing a process list is not using the credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed5h 37m into the runtook 4m 37s
startup 7.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 66.2 s plus 0.2 s for the first request, 53.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2error9h 6m into the runtook 15m 52sfree (stopped before any GPU time)
The self-check could not finish (ConnectionError: [Errno -3] Temporary failure in name resolution).
- 3passed9h 30m into the runtook 2m 23s
startup 7.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.3 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 62.3 s plus 0.1 s for the first request, 56.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 10:43 UTC. Final solution summary:
What was built
A Qwen2.5-Coder-1.5B fine-tuned binary classifier that predicts Lean 4 file validity in a single forward pass (loss on one answer token: " Yes"/" No" after a fixed question prompt), served in bf16 with a logit-difference threshold of 0.0.
Results
- Diagnostic macro F1 on the 526 provided attempts: 0.743 (memorization-optimal threshold would have given 0.757, but held-out subsets favored 0.0, and the eval is held-out attempts)
- Selfcheck: PASSED all gates — startup 7.3s (of 300), 56.7 attempts/s at 64 concurrency (needs 5.9), 0 failed requests, no-Lean scan clean
Training data (all Lean-verified ground truth)
45,120 class-balanced examples: Goedel-Prover-SFT + DeepSeek-Prover-V1.5-RL attempts (vLLM) on statements from Lean-workbook-proofs / Lean-Workbook / Goedel-Pset-v1 train shards, statement mutations, small proof edits, re-verified corpus files, and the 526 permitted validation attempts. Labels from actual Lean 4.15.0 + Mathlib v4.15.0 compilation — the pipeline reproduced all 526 provided labels (100%).
Key selection decisions
- 1.5B beat the 7B on every metric (526 F1 0.743 vs 0.719, held-out val 0.790 vs 0.762) despite identical training; a logit-average ensemble was also worse
- Threshold 0.0 chosen because the two held-out proxies (new attempts on held-out theorems) preferred it, while the in-sample preference for -0.5 reflected memorization
- Grouped validation split by statement prevented leakage in selection