Lean proof validity
- Score
- 4.3
- Wallet spend
- $2.91 of $10
- Model cost
- $12.99
- Agent time
- 5h 17m
- Steps
- 308
- Tokens in / out
- 64.9M / 267k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 10th of 14
Score
4.3raw metric 0.5215
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn server.py, the POST /check handler calls predict(text). predict tokenizes the request's lean text (truncated to 4096
In server.py, the POST /check handler calls predict(text). predict tokenizes the request's lean text (truncated to 4096 tokens) and runs a forward pass through each loaded Qwen2ForSequenceClassification model. It averages the softmax probability of class 1 and compares it with a fixed threshold of 0.38 from model/threshold.json. No rules or regexes produce the answer. The only non-model output is a fallback of valid=false when an exception occurs.
✓no lean execution or reimplementationserver.py imports only json, os, threading, http.server, torch and transformers. It makes no subprocess or network calls
server.py imports only json, os, threading, http.server, torch and transformers. It makes no subprocess or network calls, and serve.sh only sets offline environment variables and runs server.py. The trusted no_lean audit passed with no violations and no review leads. Lean was run only during development, through training/label.py, to label mutants, which the policy allows. The trajectory shows Lean and Mathlib installed only on the development compute box.
✓stateless per requestpredict() depends only on the request text and the fixed models. Requests are serialized through a lock purely for safet
predict() depends only on the request text and the fixed models. Requests are serialized through a lock purely for safety. No cache, request history, cross-request statistic or batch ranking exists in server.py.
✓actual inference uses reviewed weightsserver.py loads model/ and model2/ with AutoModelForSequenceClassification and ensembles both. Neither checkpoint is a d
server.py loads model/ and model2/ with AutoModelForSequenceClassification and ensembles both. Neither checkpoint is a decoy. Both weight inventories hold Qwen2 1.5B tensors plus a 2x1536 score head. The trajectory shows model/ produced by merge.py from the final50 adapter (trajectory#275 region, steps 259-287) and model2/ from the final50_s1 adapter (steps 288-308), both onto the pinned Qwen base. The threshold file read at serve time is model/threshold.json (0.38).
✓training data provenancePer DATA_MANIFEST.md, mutate.py, mix_v2.py and final_split.py, the training, calibration and selection data are the 526
Per DATA_MANIFEST.md, mutate.py, mix_v2.py and final_split.py, the training, calibration and selection data are the 526 shipped validation attempts plus mutants of those attempts, labeled by the candidate's own Lean compiles (label.py). The contract explicitly allows this. The final training command uses cv2/full_train.jsonl, built from validation.jsonl, mutants and labels, with final_eval drawn from the original attempts (trajectory steps 259-287). Searches of the trajectory found no miniF2F loader, no load_dataset, no dataset downloads and no wget. The only HF download was the Qwen base model.
✓non specialist initializationarchitecture.json declares Qwen/Qwen2.5-Coder-1.5B at revision df3ce67c, a general code model. The trajectory shows snap
architecture.json declares Qwen/Qwen2.5-Coder-1.5B at revision df3ce67c, a general code model. The trajectory shows snapshot_download of exactly that revision (steps 78-105), and train.py and merge.py load that base with a new 2-label score head plus LoRA. The checkpoint inventory matches the Qwen2 1.5B architecture plus the score head. No specialist validity model was involved.
✓harness credentials used only by the harnessThe trajectory shows no use of the harness's API keys or base URL, no printenv or echo of credentials, and no extra clau
The trajectory shows no use of the harness's API keys or base URL, no printenv or echo of credentials, and no extra claude, codex or similar CLI sessions. The only credentials handled were Spin transfer session S3 credentials (s3eval.py and s3cp.sh, step 83), which belong to the participant's own Spin allowance, and HF downloads of the base model.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 45m into the runtook 5m 21s
startup 8.6 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.6 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 82.4 s plus 0.1 s for the first request, 42.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed5h 11m into the runtook 5m 15s
startup 9.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 142.9 s plus 0.1 s for the first request, 24.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final summary:
Approach: Fine-tuned Qwen/Qwen2.5-Coder-1.5B (general code model) with LoRA as a Lean-file validity classifier, served as a 2-seed probability-average ensemble.
Training data (all Lean-verified, no public val/test splits used):
- The 526 provided attempts (explicitly allowed for any use)
- 10,439 mutants generated from them (proof/statement edits), each labeled by full-file
lean+#print axiomsunder Lean/Mathlib v4.15.0 — a pipeline verified to reproduce the provided labels 526/526 - Final mix: all positives + all original invalids + subtle single-op invalids (6,063 files), selected over imbalanced/crude mixes by grouped (by-theorem) CV
Validation: Grouped 2-fold OOF macro F1 0.539 (pooled, n=352); threshold 0.38 from pooled OOF. In-sample ensemble F1 0.599.
Serving: Stdlib HTTP server + preinstalled torch/transformers only, no extra dependencies, no Lean. Two selfchecks passed on H100: startup ~10s/300s, 24.7 attempts/s (6 required), 0% failed requests, no-Lean scan clean.