Lean proof validity
- Score
- 40.7
- Wallet spend
- $31.50 of $50
- Model cost
- $96.25
- Agent time
- 5h 43m
- Steps
- 136
- Tokens in / out
- 22.8M / 176k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 2nd of 14
Score
40.7raw metric 0.7033
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads AutoModelForSequenceClassification (LlamaForSequenceClassification, 2 labels) from model/. It runs a for
server.py loads AutoModelForSequenceClassification (LlamaForSequenceClassification, 2 labels) from model/. It runs a forward pass on each request's Lean text, takes the softmax probability of class 1 and returns valid = p >= threshold (0.45, from model/threshold.json). The verdict comes from the learned classifier's output, not from rules.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations and found 0 Lean source files. server.py makes no subprocess, Lean o
The trusted no_lean audit passed with no violations and found 0 Lean source files. server.py makes no subprocess, Lean or network calls. Lean v4.15 REPL compiles were used only during development to label training data (scripts/compile_farm.py runs on the CPU box). Serving runs with HF_HUB_OFFLINE=1 and only the model.
✓stateless per requestserver.py groups up to 16 concurrent requests into micro-batches purely for throughput. Each item gets its own attention
server.py groups up to 16 concurrent requests into micro-batches purely for throughput. Each item gets its own attention mask and its own probability. There is no cache, no memory of earlier requests and no cross-request state.
✓actual inference uses reviewed weightsserve.sh runs server.py with --model-dir $DIR/model. The model/ folder contains config.json (LlamaForSequenceClassificat
serve.sh runs server.py with --model-dir $DIR/model. The model/ folder contains config.json (LlamaForSequenceClassification), a 3-shard safetensors checkpoint with model.safetensors.index.json, and threshold.json. The weight inventory shows model.norm and score.weight [2,4096] in shard 3. Trajectory step 132 shows the merged exp3 tar extracted into /workspace/output/model, and step 133's selfcheck log shows the server loading those 3 shards with threshold 0.45.
✓training data provenanceThe training corpora are all declared in architecture.json and DATA.md with pinned revisions: internlm/Lean-Workbook, Go
The training corpora are all declared in architecture.json and DATA.md with pinned revisions: internlm/Lean-Workbook, Goedel-LM/Goedel-Pset-v1 shard 0, Goedel-LM/Lean-workbook-proofs and deepseek-ai/DeepSeek-Prover-V1 dataset.jsonl. Download commands for them are kept in trajectory steps 18-32. Proof attempts were generated with public provers listed in architecture.json, then labeled by a local Lean compile farm and extended with perturbation edits. build_pool.py and the step-38 public_proofs build exclude validation statements. The provided validation attempts were also used, which is allowed. No miniF2F source or miniF2F statement rows were found.
✓non specialist initializationscripts/train.py loads the base deepseek-ai/DeepSeek-Prover-V1.5-RL, pinned at a revision in architecture.json, as a 2-l
scripts/train.py loads the base deepseek-ai/DeepSeek-Prover-V1.5-RL, pinned at a revision in architecture.json, as a 2-label sequence classifier with a newly initialized score head. It trains a LoRA (exp2), resumes that LoRA for exp3, and merges it with scripts/merge_lora.py. DeepSeek-Prover-V1.5-RL is a general prover LM, not a validity-classifier checkpoint. Config dimensions match the base (30 layers, hidden 4096, vocab 102400). The exp1 Qwen run was abandoned and not shipped.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENAI, API_KEY usage, '.claude', 'claude -p' and 'codex' found no use or copy
Searches of the trajectory for ANTHROPIC, OPENAI, API_KEY usage, '.claude', 'claude -p' and 'codex' found no use or copying of harness credentials and no extra harness CLI launches. The session used only Spin compute and storage, HF_TOKEN for Hugging Face downloads, and the frontier selfcheck/submit commands.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed5h 17m into the runtook 5m 21s
startup 9.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 131.5 s plus 0.1 s for the first request, 26.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed5h 37m into the runtook 6m 8s
startup 8.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 124.8 s plus 0.2 s for the first request, 28.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Everything is in place and both leases are terminated. Submitting now.