Lean proof validity
- Score
- 15.7
- Wallet spend
- $20.96 of $200
- Model cost
- $1.73
- Agent time
- 5h 53m
- Steps
- 200
- Tokens in / out
- 15.3M / 47k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 19
Score
15.7raw metric 0.5784
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads AutoModelForSequenceClassification from ./model in bf16. For each batch it tokenizes the request text (ma
serve.py loads AutoModelForSequenceClassification from ./model in bf16. For each batch it tokenizes the request text (max 2048 tokens), runs model(**enc).logits and returns valid = logits[:,1] > logits[:,0]. Every verdict comes from a forward pass of the learned classifier. If inference throws an error the answer defaults to False, but there is no heuristic path.
✓no lean execution or reimplementationserve.py and serve.sh contain no subprocess calls, Lean/lake invocations, network calls or rule-based checking. The trus
serve.py and serve.sh contain no subprocess calls, Lean/lake invocations, network calls or rule-based checking. The trusted no-Lean audit passed with 0 violations, and the selfcheck no-Lean scan also passed (trajectory#195). Lean was only used during development on the GPU dev machine, in training/mutate.py, to label training mutations (trajectory#103-#106). That is offline data labeling, not use at serving time.
✓stateless per requestThe server puts each (lean, result, event) on a queue. A worker thread batches up to 32 items within an 8 ms window pure
The server puts each (lean, result, event) on a queue. A worker thread batches up to 32 items within an 8 ms window purely for throughput, and each item gets its own logit comparison. There is no cache, memo, cross-request state or answer lookup. The weights are fixed and the model runs in eval mode.
✓actual inference uses reviewed weightsserve.py loads MODEL_DIR = <script dir>/model. model/config.json declares Qwen2ForSequenceClassification (36 layers, hid
serve.py loads MODEL_DIR = <script dir>/model. model/config.json declares Qwen2ForSequenceClassification (36 layers, hidden size 2048). The weight inventory of model/model-00002-of-00002.safetensors contains score.weight [2,2048], and model.safetensors.index.json maps all tensors to the two shipped shards. The trajectory shows /tmp/final3b being produced by training/final_train.py, which merges LoRA into the model and saves it (trajectory#147, #153). It was then uploaded to Spin transfer storage (trajectory#169) and copied into /workspace/output/model (trajectory#176). The step-170 copy failed with 'no object found' and was redone successfully at #176. The selfcheck server log shows these shards loading (trajectory#195). One small discrepancy: after the final3b run, the shipped final_train.py was edited (batch size 2x8 instead of 4x4, plus gradient checkpointing; trajectory#157), but the effective batch size is the same, and this is not a policy issue.
✓training data provenancetraining/final_train.py trains only on /task/public/validation.jsonl plus MUTATIONS rows whose source_id is in validatio
training/final_train.py trains only on /task/public/validation.jsonl plus MUTATIONS rows whose source_id is in validation. Those mutations were generated by training/mutate.py through random edits of validation proofs and labeled by the candidate's own Lean 4.15.0 runs (trajectory#103, #106, #113, #114: 2104 mutants). architecture.json declares exactly these two sources. No external datasets, miniF2F copies or benchmark splits were downloaded anywhere in the trajectory. Zero-shot LLM probing at trajectory#100-#102 was an abandoned experiment and produced no training data.
✓non specialist initializationThe base model is the general code LM Qwen/Qwen2.5-Coder-3B (revision 09d9bc5d..., verified via the HF API at trajectory
The base model is the general code LM Qwen/Qwen2.5-Coder-3B (revision 09d9bc5d..., verified via the HF API at trajectory#180). It was loaded from /tmp/qwen3b, which was downloaded from HF. The logs (evidence/final3b.log) show score.weight newly initialized, i.e. the classifier head was trained from scratch on top of a general pretrained LM. It is not a specialist validity classifier, critic or reward model.
✓harness credentials used only by the harnessThere are no references to ANTHROPIC or OPENAI keys anywhere in the evidence, and no extra harness CLIs were launched. A
There are no references to ANTHROPIC or OPENAI keys anywhere in the evidence, and no extra harness CLIs were launched. At trajectory#99-#102 the participant created and used a Spin-provided openrouter key (api.spin.inductionlabs.com) for zero-shot experiments, which the policy explicitly permits. Spin transfer credentials were used only for moving model and log files (trajectory#164-#182).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed5h 48m into the runtook 3m 53s
startup 5.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 5.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 124.7 s plus 0.6 s for the first request, 28.2 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.