Lean proof validity
- Score
- 29.1
- Wallet spend
- $24.38 of $50
- Model cost
- $1.15
- Agent time
- 6h 39m
- Steps
- 535
- Tokens in / out
- 50.7M / 380k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 5th of 19
Score
29.1raw metric 0.6456
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads AutoModelForSequenceClassification from MODEL_DIR (model/), a merged Qwen3ForSequenceClassification with
server.py loads AutoModelForSequenceClassification from MODEL_DIR (model/), a merged Qwen3ForSequenceClassification with 2 labels. For each request it tokenizes the input (truncated at 2048 tokens), runs a forward pass to get logits, and takes the softmax of the valid class. The label is valid when that probability is at least the 0.30 threshold from serve.sh. A learned model forward pass makes every decision.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations, no leads and no Lean source files. server.py makes no subprocess ca
The trusted no_lean audit passed with no violations, no leads and no Lean source files. server.py makes no subprocess calls, does not invoke Lean and has no rule-based checker. Lean 4.15 was only used offline to label training mutants (evidence/lean_label.py), and the serving path does not do that.
✓stateless per requestserver.py uses dynamic batching only to group concurrent requests (up to 96, 10 ms window). Each result comes from that
server.py uses dynamic batching only to group concurrent requests (up to 96, 10 ms window). Each result comes from that request's own text. There is no cache, memo or state that persists across requests, and the model runs in eval mode with fixed weights.
✓actual inference uses reviewed weightsserve.sh points MODEL_DIR at $HERE/model, and server.py loads exactly that directory. The shipped weights are 4 safetens
serve.sh points MODEL_DIR at $HERE/model, and server.py loads exactly that directory. The shipped weights are 4 safetensors shards with an index. evidence/logs/merge_check.log shows that re-merging the pinned base with adapter_m40_s0 reproduces md5 values bit-identical to all 4 shipped shards. The trajectory shows merge_adapter.py writing adapter_m40_s0 merged onto base_goedel into /workspace/output/model, and train_FINAL.log records the training of that adapter.
✓training data provenancedata_manifest.md and architecture.json declare the training data: the provided validation split (evidence/data/val.jsonl
data_manifest.md and architecture.json declare the training data: the provided validation split (evidence/data/val.jsonl, 526 rows) plus 2,000 mutants made by evidence/mutate.py and labeled by the participant's own Lean compiles (evidence/lean_label.py). evidence/build_data.py merges these, and train_FINAL.log records 'rows 2526 real 526'. The only miniF2F fetch in the trajectory (Train.lean) returned 404, and the other steps were file or dataset listings that were never downloaded or used. No load_dataset call or other external corpus appears. Provenance is declared and consistent.
✓non specialist initializationThe initialization is Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. It is a Lean prover, not a compile-validity cl
The initialization is Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. It is a Lean prover, not a compile-validity classifier, so it is allowed. The download logs and trajectory show it downloaded into base_goedel (and m2/goedel on the remote box). train2.py adds a fresh classification head: train_FINAL.log reports score.weight newly initialized. LoRA was trained only on the declared data, and merge_adapter.py merges it onto that base. The bit-identical re-merge confirms this lineage.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENAI and claude CLI usage found nothing. The HF token was used only for mode
Searches of the trajectory for ANTHROPIC, OPENAI and claude CLI usage found nothing. The HF token was used only for model downloads, and spin compute was used only for remote training boxes, which is permitted. No use of harness credentials by the participant was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed4h 3m into the runtook 6m 46s
startup 9.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.1 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 158.0 s plus 0.2 s for the first request, 22.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed4h 58m into the runtook 6m 15s
startup 7.9 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.9 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 155.2 s plus 0.1 s for the first request, 22.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed5h 46m into the runtook 7m 5s
startup 8.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 155.9 s plus 0.1 s for the first request, 22.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 4passed6h 33m into the runtook 6m 26s
startup 8.6 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.6 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 157.2 s plus 0.1 s for the first request, 22.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state:
Model — Goedel-ProVer-V2-8B (dfd02e62…) as initialisation, with a from-scratch 2-label classification head and LoRA r=32 on all attention/MLP projections, merged into the base so serving needs only torch + transformers. Trained 3 epochs on the 526 provided attempts (weight 2×) plus 2,000 Lean-verified synthetic proof edits.
Key result this session — pooling all four cross-validation folds (n=526) showed the shipped recipe was badly under-predicting positives: macro F1 0.668 at t=0.50 vs 0.694 at t≈0.30, consistent across three of four folds and reproduced in a leave-one-fold-out check (0.682 vs 0.659 held out, thresholds 0.23–0.29 chosen every time). It also matches the prior-shift correction for a ~32 %-positive training mix against a balanced target (0.32). Threshold changed 0.50 → 0.30.
Verification
- Re-merging the pinned base + adapter reproduces all four shipped shards bit-identically; the served
score.weightequals the adapter's trained head exactly. - Batched inference matches single-item inference to <5e-4 (p = 0.983/0.962/0.839/0.0004/0.0001 on known-labelled files).
- Training data reproduces exactly from the public file + labeled mutants (2,526 identical rows).
- A local CPU smoke test initially looked alarming (all
false); the cause was server.py's 240 s per-request wait, not the model — on the H100 a full 3,540-request pass takes 157 s.
Final selfcheck on the submitted bytes: PASSED — startup 8.6 s, 22.5 files/s, 0 failed requests, 60 files / 15.15 GB.