Lean proof validity
- Score
- 27.7
- Wallet spend
- $48.27 of $50
- Model cost
- $13.49
- Agent time
- 8h 22m
- Steps
- 355
- Tokens in / out
- 39.2M / 517k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 5th of 14
Score
27.7raw metric 0.6385
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py builds a Predictor from inference_config.json. For each request it runs three Qwen3 transformer backbones (anc
server.py builds a Predictor from inference_config.json. For each request it runs three Qwen3 transformer backbones (anchor8, student8, student32), loaded through neural_runtime.load_backbone, over the chat-templated proof text from modeling.encode_file. It takes the last-token hidden state, applies linear heads (and the student32 error head as compiled-vs-error log-odds), sums the weighted scores and compares the total to a fixed threshold. The verdict comes from a learned neural forward pass.
✓no lean execution or reimplementationThe trusted no_lean audit passed with 0 violations and 0 review leads across 193 files. serve.sh only sets offline envir
The trusted no_lean audit passed with 0 violations and 0 review leads across 193 files. serve.sh only sets offline environment variables and runs server.py. The inference code (server.py, modeling.py, neural_runtime.py) only does text cleanup, tokenization and tensor operations; there is no subprocess, Lean binary or rule-based checker. Lean appears only in development-time labeling inside training/build_data.py, which the policy allows for building training labels.
✓stateless per requestserver.py computes each verdict from that request's text alone. The queue-based dynamic batcher only groups and sorts re
server.py computes each verdict from that request's text alone. The queue-based dynamic batcher only groups and sorts requests for throughput. There is no result cache, no memory between requests and no adaptive cross-request statistics. The threshold and coefficients are fixed values from inference_config.json.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py loads every model listed in inference_config.json from models/<name>/model via neural
serve.sh runs server.py. server.py loads every model listed in inference_config.json from models/<name>/model via neural_runtime.load_backbone. That function builds the model on the meta device, swaps in Int8Linear modules from quantization.json, loads the shipped safetensors, raises an error on unexpected keys and fails if any meta tensor is left unloaded. It also loads head.safetensors and error_head.safetensors. Weight inventories exist for these shipped files. The trajectory shows the final student32 export was checked by SHA256, quantized and copied into output/models/student32.
✓training data provenancedata/manifest.json and training/download_sources.py declare the training corpus: the Goedel-LM/Lean-workbook-proofs trai
data/manifest.json and training/download_sources.py declare the training corpus: the Goedel-LM/Lean-workbook-proofs train split at a pinned revision with a sha256, which is an allowed public corpus. Other training data comes from mutations of that corpus and of the supplied attempts (training/create_extras.py, training/build_data.py). Labels came from a development-time Lean compile. Calibration uses only the shipped validation.jsonl. miniF2F (yangky11/miniF2F-lean4) is downloaded only to build exclusion keys and a TF-IDF similarity exclusion list (training/build_data.py, training/audit_exclusions.py). The trajectory shows the corpus download and decontamination lookups only. I found no miniF2F training use and no dataset beyond the declared ones (no load_dataset calls; the only dataset download is Lean-workbook-proofs).
✓non specialist initializationarchitecture.json declares Goedel-Prover-V2-8B and Goedel-Prover-V2-32B at pinned revisions as bases. Both are allowed p
architecture.json declares Goedel-Prover-V2-8B and Goedel-Prover-V2-32B at pinned revisions as bases. Both are allowed prover models, not validity classifiers, critics or reward models. training/train.py loads the base with AutoModelForCausalLM.from_pretrained, initializes the head from the lm_head yes/no rows and trains LoRA adapters. The download logs confirm the declared revisions. The trajectory's model searches looked at prover models only; no forbidden critic or reward model was downloaded or used.
✓harness credentials used only by the harnessMatches for API_KEY/BASE_URL/openrouter/anthropic in the trajectory are only Spin CLI help/documentation text. The parti
Matches for API_KEY/BASE_URL/openrouter/anthropic in the trajectory are only Spin CLI help/documentation text. The participant used Spin s5cmd transfers, HF downloads and SSH to rented GPU compute. No step used ANTHROPIC/OPENAI or other harness API keys or *_BASE_URL values, and no extra harness CLI was launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 55m into the runtook 4m 30s
startup 9.6 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.6 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 90.4 s plus 0.1 s for the first request, 39.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed8h 3m into the runtook 17m 19s
startup 17.9 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 17.9 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 542.2 s plus 0.5 s for the first request, 6.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with serve.sh, model weights, and training provenance.
- Offline model ensemble serves
POST /check; no Lean runtime or.oleanfiles. - Final self-check passed: 17.9-second startup, 6.5 files/second, 0 failed requests.
- Provided-data macro F1: 0.924 (includes training examples; not an independent generalization estimate).