Lean proof validity
- Score
- 0.0
- Wallet spend
- $22.28 of $50
- Model cost
- $39.82
- Agent time
- 7h 45m
- Steps
- 601
- Tokens in / out
- 57.8M / 342k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 11th of 14
Score
0.0raw metric 0.4806
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads AutoTokenizer and AutoModelForSequenceClassification from ./model in bf16 on CUDA. For each request it r
server.py loads AutoTokenizer and AutoModelForSequenceClassification from ./model in bf16 on CUDA. For each request it runs a forward pass, takes softmax(logits)[:,1] as P(valid) and compares it with the learned threshold 0.2 from model/threshold.json. model/config.json declares Qwen2ForSequenceClassification. The verdict comes from the model's output, not from rules.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations or review leads. The serving path (serve.sh, then server.py) has no
The trusted no_lean audit passed with no violations or review leads. The serving path (serve.sh, then server.py) has no subprocess, no Lean or lake invocation, and no rule-based checker. Lean 4.15 and Mathlib were used only offline during development to label training data. They are not part of serving.
✓stateless per requestserver.py has no cache, no memo of earlier inputs and no cross-request statistics. Requests are batched and sorted by le
server.py has no cache, no memo of earlier inputs and no cross-request statistics. Requests are batched and sorted by length only for throughput. Left padding is masked with an attention mask, so each prediction depends only on its own text. Each POST /check returns {valid: bool} for that request alone.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=./model and runs server.py, which loads the shipped model directory. That directory contains mod
serve.sh sets MODEL_DIR=./model and runs server.py, which loads the shipped model directory. That directory contains model.safetensors.index.json and three shards; shard 3 holds the score.weight [2,3584] classification head. The trajectory shows the stage-2 model served through this serve.sh, validated at macro F1 0.834 with t=0.20, and extracted into the output model directory (trajectory#584, trajectory#589).
✓training data provenancedownloads/DATA_MANIFEST.md and download_commands.sh declare every source: - public train splits of internlm/Lean-Workboo
downloads/DATA_MANIFEST.md and download_commands.sh declare every source: - public train splits of internlm/Lean-Workbook, Goedel-LM/Goedel-Pset-v1 and Goedel-LM/Lean-workbook-proofs - attempts generated with Goedel-Prover-SFT - labels produced locally with Lean - the provided /task/public/validation.jsonl (trajectory steps-259-292) miniF2F (cloned from yangky11/miniF2F-lean4, trajectory#91-92) was used only to remove overlapping statements, in build_pool.py and prepare_lwb.py. build_focus.py splits the provided rows by hash and holds half out for threshold selection. The download commands are retained, but there are no download logs beyond the trajectory, so those downloads are recorded as unverified. No undeclared corpus was found.
✓non specialist initializationarchitecture.json declares Qwen/Qwen2.5-Coder-7B at a pinned revision as the base. train_clf.py uses this base by defaul
architecture.json declares Qwen/Qwen2.5-Coder-7B at a pinned revision as the base. train_clf.py uses this base by default and applies LoRA SEQ_CLS, followed by merge_and_unload. train_box4.log shows score.weight newly initialized from Qwen/Qwen2.5-Coder-7B, and stage 2 (focus_train.log) continues from the stage-1 checkpoint. The trajectory shows the Qwen base being downloaded from the Hub. It is a general code model, not a Lean-validity specialist.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, API_KEY, BASE_URL, openrouter, claude and printenv found no use of harness cre
Searches of the trajectory for ANTHROPIC, API_KEY, BASE_URL, openrouter, claude and printenv found no use of harness credentials. The only credentials in use were the participant's own Spin compute session and the transfer tooling (xfer.py with the Spin session). The shipped serving code needs no external API credentials and runs with Hugging Face offline mode.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed6h 57m into the runtook 4m 46s
startup 8.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.1 s
- ✓POST /check · {"id": "v00000", "valid": true}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 77.0 s plus 0.2 s for the first request, 45.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed7h 39m into the runtook 5m 19s
startup 10.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 10.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 85.8 s plus 0.2 s for the first request, 41.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-06 10:54:19 UTC. The session is complete.
Final solution summary
Model: Qwen2.5-Coder-7B → Qwen2ForSequenceClassification (2-class head), two-stage LoRA fine-tuning (r=32, merged weights), decision rule P(valid) > 0.20.
Data pipeline (all provenance in downloads/DATA_MANIFEST.md):
- Lean-workbook-proofs + Goedel-Prover-SFT-generated attempts, re-labeled by actually running Lean 4.15.0 + Mathlib v4.15.0 (93%/22% valid) → 21,036 balanced training rows
- Task-provided 526 validation samples (explicitly permitted), hash-split: half oversampled ×8 into stage 2, half held out for honest measurement
- miniF2F used only for decontamination
Key fix this session: the stage-1 model scored 0.92 on held-out pool data but only 0.62 on the provided distribution — it confidently over-predicted "valid" on near-miss tactic failures (linarith/unknown-identifier/unsolved-goals errors). The stage-2 focused continuation closed part of that gap: 0.645 → 0.691 on the clean held-out provided half, with pool val unchanged (0.9167).
Verification on the real eval infra (both selfchecks passed):
- Startup: 10.2s of 300s
- Throughput: 41.1 attempts/s at concurrency 64 vs 5.9 required (7× margin)
- Failed requests: 0 of 3540
- no-Lean scan and prerequisites: passed
- Public diagnostic through
serve.sh: 0.834 macro F1, 0 failed requests