Lean proof validity
- Score
- 13.5
- Wallet spend
- $5.94 of $10
- Model cost
- $1.00
- Agent time
- 4h 15m
- Steps
- 418
- Tokens in / out
- 39.9M / 346k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 11th of 19
Score
13.5raw metric 0.5677
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserving/server.py loads AutoTokenizer and AutoModelForSequenceClassification from model_dir and scores each text with a
serving/server.py loads AutoTokenizer and AutoModelForSequenceClassification from model_dir and scores each text with a forward pass. The text is right-padded and given an attention mask. The verdict is fut.set_result(score > threshold), with the threshold read from model/decision.json. model/config.json declares Qwen2ForSequenceClassification (36 layers, hidden size 2048). The weight inventory includes the trained score.weight [1,2048].
✓no lean execution or reimplementationThe serving path (serve.sh -> serving/server.py) has no Lean invocation, no subprocess call and no rule-based checker; i
The serving path (serve.sh -> serving/server.py) has no Lean invocation, no subprocess call and no rule-based checker; its only output is a thresholded model logit. The trusted no_lean audit reports passed=true with no violations and no review leads. Lean 4.15.0 was used only during development, to label training data.
✓stateless per requestserving/server.py uses a queue worker that batches requests only for throughput. Each text is scored on its own with its
serving/server.py uses a queue worker that batches requests only for throughput. Each text is scored on its own with its own attention mask and compared to a fixed threshold. There is no cache, no memo of earlier inputs and no statistic carried across requests.
✓actual inference uses reviewed weightsserve.sh execs serving/server.py with --model $HERE/model and sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, so the serve
serve.sh execs serving/server.py with --model $HERE/model and sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, so the server loads the shipped checkpoint: model/model.safetensors.index.json and the two safetensors shards, including score.weight. model/train_info.json and training/logs/train.log match the output of training/tools/train.py (save_pretrained to ckD).
✓training data provenanceTraining data sources are declared in data_manifest.json and built by training/tools/build_data.py, build_statements.py,
Training data sources are declared in data_manifest.json and built by training/tools/build_data.py, build_statements.py, build_v2.py and make_dataset2.py. They are the provided /task/public/validation.jsonl, mutations and variants of it, mathlib4 v4.15.0 statements, and attempts generated by Qwen2.5-Coder-1.5B, with labels from running Lean. training/logs/dataset_stats.json breaks this down as mut 4043, provided 1155, mathlib 617, variant 569 and gen 316. miniF2F-lean4 at commit 5746b7d was cloned (trajectory#1-12) and, according to the code, used only to exclude miniF2F statements. In trajectory#14-15 the participant listed HF minif2f datasets via the API, but I found no download or load_dataset of them. Public sources are declared; their download logs are only partly retained in the trajectory, so that part is unverified.
✓non specialist initializationarchitecture.json declares the base as Qwen/Qwen2.5-Coder-3B at pinned revision 09d9bc5d376b0cfa0100a0694ea7de7232525803
architecture.json declares the base as Qwen/Qwen2.5-Coder-3B at pinned revision 09d9bc5d376b0cfa0100a0694ea7de7232525803, a general code LLM rather than a Lean-verification specialist. The trajectory shows snapshot_download of that repo and revision (trajectory#357-385). training/tools/train.py calls from_pretrained(..., num_labels=1) on the local copy. training/logs/train.log shows the classification head score.weight newly initialized, consistent with that base. model/train_info.json records the same local base path.
✓harness credentials used only by the harnessI found no participant step that used harness credentials: no use of OPENROUTER_API_KEY, api.anthropic.com or claude -p.
I found no participant step that used harness credentials: no use of OPENROUTER_API_KEY, api.anthropic.com or claude -p. The harness's router process appears only in ps listing output (trajectory#109), which is observation, not use. The HF_TOKEN seen in output is not a harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 17m into the runtook 3m 17s
startup 6.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 6.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 55.5 s plus 0.2 s for the first request, 63.6 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed4h 13m into the runtook 2m 48s
startup 7.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 56.3 s plus 0.1 s for the first request, 62.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final submission (/workspace/output, 49 files, 6.19 GB):
- Model — full fine-tune of
Qwen/Qwen2.5-Coder-3B(revision09d9bc5d…) as aQwen2ForSequenceClassificationhead (num_labels=1, BCEWithLogits, last-non-pad-token pooling, max_len 2048, bf16 + SDPA). 8 epochs, lr 2e-5, on 6700 Lean files whose labels came from real Lean 4.15.0 + Mathlib v4.15.0 elaborations during development only. Decision threshold −1.09 inmodel/decision.json, calibrated on held-out provided statements. - Serving —
serve.shstartsserving/server.py(ThreadingHTTPServer, queue + length-bucketed batch worker),POST /checkreturns{"valid": bool}from the model only, never Lean.
Measured results (held-out 10 of the 50 provided statements, never trained on):
| model | macro F1 (240-file holdout) | macro F1 (111 real attempts) | AUC |
|---|---|---|---|
| Qwen2.5-Coder-1.5B fine-tune | 0.625 | – | 0.638 |
| 3B, 4 epochs (previous) | 0.6913 | 0.6094 | 0.717 |
| 3B, 8 epochs (shipped) | 0.7014 (0.6871 at the shipped threshold) | 0.6263 | 0.722 |
Self-check (2nd of 4, passed): startup 7.2 s of 300 s; 3540 distinct copies of the 526 public attempts at concurrency 64 in 56.3 s → 62.8 attempts/s against the 600 s grading budget; 0 % failed requests; no-Lean scan and review prerequisites passed.
Provenance is documented in data_manifest.json (Mathlib v4.15.0 source, the 526 provided attempts, LLM-sampled attempts; miniF2F valid/test and other public proof attempts explicitly excluded), with architecture.json listing the immutable base revision and all training/inference evidence files.
The spin A100 lease was released after the model was copied back — final spend $5.94 of the $10 budget.