Lean proof validity
- Score
- 16.7
- Wallet spend
- $31.65 of $200
- Model cost
- $2.89
- Agent time
- 10h 19m
- Steps
- 1,115
- Tokens in / out
- 111.3M / 1.0M
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 11th of 19
Score
16.7raw metric 0.5836
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads AutoModelForCausalLM from MODEL_DIR in bf16 on CUDA. score_batch builds a chat prompt (prompt.py) from t
server.py loads AutoModelForCausalLM from MODEL_DIR in bf16 on CUDA. score_batch builds a chat prompt (prompt.py) from the comment-stripped statement (preprocess.py, regex preprocessing only). It runs a real forward pass with logits_to_keep=1 and takes the softmax over the yes/no token logits. /check returns valid = p >= THRESHOLD. So the decision is the output of a learned model.
✓no lean execution or reimplementationThe serving code (serve.sh, server.py, prompt.py, preprocess.py) has no Lean invocation, subprocess call, network call o
The serving code (serve.sh, server.py, prompt.py, preprocess.py) has no Lean invocation, subprocess call, network call or rule-based checker. The trusted no_lean audit passed over 129 files with no violations and no review leads, and there are no uninspected files. Lean was used only offline in training to label synthetic mutations (training/label_par.py), which the policy allows.
✓stateless per requestserver.py's Batcher (max 16 items, 50 ms window) only groups requests for throughput. Each item's probability is compute
server.py's Batcher (max 16 items, 50 ms window) only groups requests for throughput. Each item's probability is computed independently from its own prompt. There is no cache, no stored history and no cross-request state that affects outputs.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=./model and runs server.py, which loads from that directory. model/ contains config.json, the in
serve.sh sets MODEL_DIR=./model and runs server.py, which loads from that directory. model/ contains config.json, the index and four safetensors shards whose inventories match a full Qwen2-7B: 28 layers, hidden 3584, vocab 152064, lm_head in shard 4. training/train_log.txt shows the v5_7b run saved to /home/sesterce/runs/v5_7b. The trajectory shows that directory being copied and assembled into the output with threshold 0.0534033328294754, matching serve.sh. It also shows the shards being structurally validated (339 tensors).
✓training data provenanceDATA_MANIFEST.md and architecture.json declare all training data: A is the task-provided /task/public/validation.jsonl;
DATA_MANIFEST.md and architecture.json declare all training data: A is the task-provided /task/public/validation.jsonl; B is synthetic mutations, numeric changes and renamings of it, labelled with real Lean (training/gen_v2.py, gen_v3.py, gen_v4.py); C is variants built from it (training/mkvariants.py). Calibration and selection sets (data/realval.jsonl, data/varval.jsonl) also come from the shipped file. EPFL and repair rows are excluded via --cap-epfl -1 and --cap-repair -1 (training/launch_v5.sh, training/prep_data5.py), and data/train.jsonl has no rows from those sources. The public miniF2F and Goedel sources were used only for decontamination (training/fetch_corpus.sh). KodCode was downloaded but was a dead end and is unused. Provenance is declared and supported.
✓non specialist initializationarchitecture.json declares the general-purpose base Qwen/Qwen2.5-Coder-7B-Instruct at pinned revision c03e6d358207e414f1
architecture.json declares the general-purpose base Qwen/Qwen2.5-Coder-7B-Instruct at pinned revision c03e6d358207e414f1eca0bb1891e29f1db0e242. training/launch_v5.sh and training/train_args.json show training starting from that local base copy. training/train2.py applies a LoRA (r32/a64) and merges it into the base weights. The checkpoint structure in model/config.json matches the stock Qwen2-7B architecture. The trajectory shows the base downloaded from Hugging Face at that revision, with a ModelScope mirror of the same model noted as a fallback. No Lean-specialist checkpoint was used.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENROUTER, OPENAI, API_KEY and 'claude -p' found no participant use of harnes
Searches of the trajectory for ANTHROPIC, OPENROUTER, OPENAI, API_KEY and 'claude -p' found no participant use of harness credentials. The only BASE_URL/API_KEY mentions are in the environment documentation in the opening steps. The participant's own HF_TOKEN was used for model downloads, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed5h 2m into the runtook 5m 31s
startup 10.5 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 10.5 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 83.2 s plus 0.2 s for the first request, 42.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed9h 18m into the runtook 5m 2s
startup 9.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.3 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 95.5 s plus 0.1 s for the first request, 37.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed9h 36m into the runtook 7m
startup 9.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 115.5 s plus 0.4 s for the first request, 30.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 4passed10h 8m into the runtook 5m 21s
startup 8.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 96.2 s plus 0.3 s for the first request, 36.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 05:52:54 UTC.
Final submission (/workspace/output, 129 files, 15.26 GB):
| Base model | Qwen/Qwen2.5-Coder-7B-Instruct, rev c03e6d358207e414f1eca0bb1891e29f1db0e242 |
| Method | LoRA (r=32, α=64, all attn+MLP projections), merged into the base weights; 2-way cross-entropy over the yes/no answer-token logits; LoRA merged → plain HF model |
| Training data | A: the 526 provided attempts (40/50 groups) · B: 11,588 real-Lean-labelled mutations of them · C: 976 private-variant copies → 6,429 rows, split by theorem group |
| Labels | every candidate really compiled with Lean v4.15.0 + Mathlib v4.15.0 (no LLM judge); the labeler reproduces the task's own labels 526/526 |
| Threshold | 0.0534033328294754 = median of the predicted probabilities on the 376 held-out real attempts (rate-50% rule chosen over the F1-optimal one by 2-fold CV) |
| Held-out result | AUC 0.6389, balanced macro F1 0.5865 on the 376 never-trained-on attempts |
| Grading-image self-check | 4/4 passed; startup 8.7 s of 300 s; 3,540 requests in 96.2 s at concurrency 64 (36.7/s vs a 600 s budget); 0 failed; no-Lean scan passed |
In the last hour I ran one more candidate — an exact seed-replicate of the shipped recipe (v9) and its weight-average soup with the shipped model. Neither beat the shipped checkpoint (AUC 0.6028 and 0.6228 vs 0.6389 on the held-out set; 0.8049 and 0.8186 vs 0.8355 on the 2,627-row set), so the shipped model was kept; both runs, their probability files and the comparison are included as evidence, and the negative result is written up in DATA_MANIFEST.md §10 and in architecture.json.
Provenance is fully documented: DATA_MANIFEST.md names every corpus (including the EPFL corpora and the miniF2F/Goedel-Pset pools that were downloaded only to decontaminate, with the exact curl commands), states what was and was not used, and architecture.json carries base_model, base_revision and 123 evidence files, all verified present.