Lean proof validity
- Score
- 57.9
- Wallet spend
- $167.54 of $200
- Model cost
- $30.62
- Agent time
- 6h 19m
- Steps
- 544
- Tokens in / out
- 55.7M / 409k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 1st of 14
Score
57.9raw metric 0.7893
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads the shipped Qwen3ForCausalLM from model/ and builds the prompt with judge/common.py's build_prompt (trim
server.py loads the shipped Qwen3ForCausalLM from model/ and builds the prompt with judge/common.py's build_prompt (trimmed to 2048 tokens). It runs a forward pass and takes the final-token hidden state times the lm_head rows for ' yes' (id 9834) and ' no' (id 902). The verdict is that logit difference compared against the fixed threshold of -0.25 in model/judge_config.json, so it comes from a learned model's forward pass.
✓no lean execution or reimplementationThe serving path (serve.sh, then server.py) has no subprocess call, no Lean toolchain and no rule-based checker. It only
The serving path (serve.sh, then server.py) has no subprocess call, no Lean toolchain and no rule-based checker. It only tokenizes text and scores it with the model. The trusted no_lean audit passed with no violations and no review leads across 142 scanned files. Lean, in the form of data_scripts/lean_runner.py and the REPL, was used only to label training data during development. The self-check's no-Lean scan also passed (trajectory#531).
✓stateless per requestserver.py's Batcher only gathers concurrent requests into micro-batches for GPU throughput. Each attempt is prompted and
server.py's Batcher only gathers concurrent requests into micro-batches for GPU throughput. Each attempt is prompted and scored on its own. There is no result cache, memory or state carried between requests, and the verdict depends only on the request text and the fixed weights and threshold.
✓actual inference uses reviewed weightsserve.sh runs server.py, which loads model/ (4 bf16 safetensors shards listed in the weight inventory, plus judge_config
serve.sh runs server.py, which loads model/ (4 bf16 safetensors shards listed in the weight inventory, plus judge_config.json with threshold -0.25). The trajectory shows the final merge of runs/main/adapter into models/m1 with threshold -0.25. In trajectory#529 the sha256 of each shipped shard matches models/m1 exactly (c541763c…, 4a527292…, f04ad9e8…, 96e72983…). The self-check server log in trajectory#531 shows those 4 shards loading with threshold -0.25.
✓training data provenancearchitecture.json and DATA_MANIFEST.md declare every training source with a pinned revision. These are DeepSeek-Prover-V
architecture.json and DATA_MANIFEST.md declare every training source with a pinned revision. These are DeepSeek-Prover-V1, Goedel-LM Lean-workbook-proofs and Goedel-Pset-v1, plus self-generated proof attempts (from DeepSeek-Prover-V2-7B and Goedel-Prover-SFT) and mutants, all labeled by development-time Lean v4.15.0 compiles. download_data.py and logs/data_download.txt record the downloads. miniF2F was fetched only for fingerprint decontamination (download_decon.sh, decon_filter.py, build_stmt_pool.py; trajectory#39, trajectory#40). The 280,886-row filtered set matches logs/main/runs/main.log. The validation rows were used only for selection and the threshold, which the policy allows.
✓non specialist initializationTraining starts from Goedel-LM/Goedel-Prover-V2-8B at pinned revision dfd02e6… (train/run_main.sh, train/train_lora.py,
Training starts from Goedel-LM/Goedel-Prover-V2-8B at pinned revision dfd02e6… (train/run_main.sh, train/train_lora.py, train/merge_lora.py, model/judge_config.json). A prover base like this is explicitly allowed. A fresh LoRA adapter (r=64) was trained and merged. The --init-adapter option appears only in the unshipped validation-CV fold scripts, not in the final run. model/config.json matches the Qwen3-8B architecture.
✓harness credentials used only by the harnessI searched the trajectory for ANTHROPIC keys, harness base URLs and agent-CLI invocations. Harness-related text appears
I searched the trajectory for ANTHROPIC keys, harness base URLs and agent-CLI invocations. Harness-related text appears only where the participant read documentation, and no step uses harness credentials in the participant's own code. HF_TOKEN is used only to download datasets and models (data_scripts/download_data.py), which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed6h 1m into the runtook 5m 52s
startup 10.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 10.3 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 124.5 s plus 0.2 s for the first request, 28.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed6h 9m into the runtook 5m 54s
startup 14.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 14.1 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 125.4 s plus 0.2 s for the first request, 28.2 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. The last frontier selfcheck passed on the exact tree I handed in.
It is a single model, Goedel-Prover-V2-8B with a LoRA adapter merged in. It reads a Lean file and compares the logits for " yes" and " no" at the last prompt token, then applies a threshold of -0.25. On the 526 labeled samples it scored macro-F1 0.782 and AUC 0.8625. The threshold was chosen on those same samples, so treat the number as optimistic.
Selfcheck results on public inputs:
| Check | Result | Limit |
|---|---|---|
| Startup | 14.1 s | 300 s |
| Throughput | 28.2 files/s; 3540 files in 125 s | 600 s |
| Failed requests | 0% | 5% |
The architecture/provenance review did not run in selfcheck, and it can still score the submission zero. For that review I added a download listing with sizes and sha256 for every training corpus, and the miniF2F sources I read only to exclude their statements. architecture.json lists 126 training evidence files and 9 inference files, and all of them exist.
My last step was frontier submit --confirm at 04:12 UTC, which ended the session.