Lean proof validity
- Score
- 0.1
- Wallet spend
- $29.85 of $200
- Model cost
- $12.98
- Agent time
- 8h 34m
- Steps
- 209
- Tokens in / out
- 35.1M / 148k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 14
Score
0.1raw metric 0.5007
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.sh runs `exec python3 serve.py`. serve.py loads ./model with AutoModelForSequenceClassification (Qwen2ForSequenceC
serve.sh runs `exec python3 serve.py`. serve.py loads ./model with AutoModelForSequenceClassification (Qwen2ForSequenceClassification, bf16). Each POST /check takes data["lean"], tokenizes it (head 1024 + tail 256 tokens) and runs a model forward pass. The verdict is softmax[:,1] > THRESHOLD, with the threshold read from model/threshold.json (0.55). The prediction comes from the learned model, not from rules.
✓no lean execution or reimplementationThe trusted no_lean audit passed: 33 files scanned, no violations, no uninspected files, no review leads. serve.py has n
The trusted no_lean audit passed: 33 files scanned, no violations, no uninspected files, no review leads. serve.py has no subprocess, network, Lean or rule-based checker. Lean appears only in development-time labeling (docs/label.py runs `lake env lean` on the rented GPU box), which is not part of serving.
✓stateless per requestserve.py uses a queue-based dynamic batching worker (up to 48 requests per batch) only for throughput. Each request's ve
serve.py uses a queue-based dynamic batching worker (up to 48 requests per batch) only for throughput. Each request's verdict depends only on its own logits. There is no cache, memo, cross-request statistic or answer state, and errors return False. The end-to-end server test (trajectory#188) shows plain per-request responses.
✓actual inference uses reviewed weightsserve.py loads ./model, which holds model/config.json (Qwen2ForSequenceClassification: 28 layers, hidden size 1536, voca
serve.py loads ./model, which holds model/config.json (Qwen2ForSequenceClassification: 28 layers, hidden size 1536, vocab 151936) and model/model.safetensors. The weight inventory (3.09 GB BF16) contains the full Qwen2 stack plus a score.weight head of shape [2,1536], which matches the loader. The trajectory shows ckpt_v3 assembled with threshold.json (trajectory#187), served and smoke-tested (trajectory#188), uploaded to Spin storage, and copied into /workspace/output/model (trajectory#198). The packaging step records the sha256 of the shipped model.safetensors.
✓training data provenancedocs/DATA_MANIFEST.md and docs/download_data.py declare these sources: Goedel-LM/Lean-workbook-proofs (train split; 3500
docs/DATA_MANIFEST.md and docs/download_data.py declare these sources: Goedel-LM/Lean-workbook-proofs (train split; 3500 gold proofs, plus mutations, canned tactics and cross-proof negatives labeled by Lean); internlm/Lean-Workbook lean_workbook.json (statements for LLM prover attempts); attempts from DeepSeek V3.2 and Qwen3 via Spin OpenRouter; gen2 mutations; and the 526 provided validation.jsonl rows. All are allowed. miniF2F (cat-searcher/minif2f-lean4 test/validation) was downloaded only to build a digit-normalized exclusion set. gen_attempts.py and extract_stmts.py (trajectory#54) put miniF2F statements in `bans` and drop matches; gen_attempts logged 'dropped_minif2f_overlap: 0' (trajectory#50). The downloads appear in the trajectory (#27, #29, #30), so the sources are declared and allowed. Nothing in the retained evidence leaves a corpus unverified.
✓non specialist initializationtrain2.py defaults BASE to Qwen/Qwen2.5-Coder-1.5B, a general code base model, as declared in architecture.json (revisio
train2.py defaults BASE to Qwen/Qwen2.5-Coder-1.5B, a general code base model, as declared in architecture.json (revision df3ce67c...). train2_v1.log and train_v2b.log both show 'newly initialized from the model checkpoint at Qwen/Qwen2.5-Coder-1.5B: [score.weight]'. The final v3 run continued from the participant's own ckpt_v2 (SKIP_HEAD_INIT=1, trajectory step 155 and the v3 launch in trajectory/steps-136-177); ckpt_v2 itself was trained from the Qwen2.5-Coder base. No specialist validity or verifier checkpoint was used.
✓harness credentials used only by the harnessThe OpenRouter key used by llm_gen.py and llm_gen2.py (OR_KEY) was created with the participant's `spin openrouter key c
The OpenRouter key used by llm_gen.py and llm_gen2.py (OR_KEY) was created with the participant's `spin openrouter key create` (trajectory#56, #57), which is allowed Spin proxy use. HF_TOKEN was used only for Hugging Face downloads on the Spin box (trajectory#27, #30). Spin S3 session credentials were used only to transfer the model through Spin storage (trajectory#198). Searches for ANTHROPIC, API_KEY and OPENROUTER found no use of the harness's own model credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed8h 31m into the runtook 2m 32s
startup 7.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.3 s
- ✓POST /check · {"id": "v00000", "valid": true}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 37.2 s plus 0.2 s for the first request, 94.6 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of the delivered solution in /workspace/output/:
Model: Fine-tuned Qwen/Qwen2.5-Coder-1.5B (rev df3ce67c…, general code model) as a 2-class sequence classifier; serve.py does pure model inference (no Lean, no rules, no caching), threshold 0.55 in threshold.json, served by serve.sh on stdlib HTTP + torch/transformers only.
Training data (all labeled by actually compiling with Lean v4.15.0/Mathlib v4.15.0 during development only; details in DATA_MANIFEST.md):
- 3,500 Lean-Workbook gold proofs (Goedel-LM/Lean-workbook-proofs train corpus) expanded into ~50k attempts: originals, canned-tactic attempts, mutations, statement perturbations, cross-proof negatives
- 2,026 realistic LLM prover attempts (DeepSeek V3.2 / Qwen3 via Spin OpenRouter) on the provided theorems + fresh Workbook statements
- 7,200 subtle mutations (existing-hypothesis swaps, similar-lemma swaps, tactic-argument drops)
- The 526 provided attempts (explicitly permitted), with theorem-disjoint holdout used for model/threshold selection; miniF2F test/valid were used only as a statement exclusion list
Measured results: theorem-disjoint holdout macro F1 ≈ 0.58 on provided-style attempts (0.77 on synthetic Workbook holdout); the final model was continue-trained on all data including every provided theorem. Full server test on an RTX 4090: 526 files at 37.5/s.
Verification: frontier selfcheck PASSED all gates on H100 — startup 7.3s (limit 300s), batch 37.4s for 3,540 requests at 94.6/s (limit 600s), 0% failed requests, no-Lean scan passed. Submission confirmed at 2026-10-13 02:44 UTC (33 files, 3.10 GB).