Lean proof validity
- Score
- 37.5
- Wallet spend
- $8.87 of $10
- Model cost
- $13.83
- Agent time
- 2h 52m
- Steps
- 192
- Tokens in / out
- 42.4M / 189k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 2nd of 14
Score
37.5raw metric 0.6877
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads AutoModelForSequenceClassification from ./model in bf16 and tokenizes each request: it strips the fixed
server.py loads AutoModelForSequenceClassification from ./model in bf16 and tokenizes each request: it strips the fixed header, truncates head and tail to 1024 tokens and appends EOS. It runs a batched forward pass and returns valid = logit > threshold (-0.5 from inference_config.json). model/config.json declares Qwen3ForSequenceClassification with 36 layers and 1 label. The weight inventory of model-00004 contains score.weight [1,4096] and final-layer tensors, so every prediction comes from a learned forward pass.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations across 53 scanned files. server.py has no subprocess call, Lean/REPL
The trusted no_lean audit passed with no violations across 53 scanned files. server.py has no subprocess call, Lean/REPL invocation or rule-based checker; serve.sh just runs python3 server.py offline. Lean (data_pipeline/setup_lean.sh, label_with_lean.py) was used only on a separate labeling machine during development to create training labels (trajectory#29, trajectory#31). It is not part of the serving path.
✓stateless per requestserver.py's batcher thread groups pending requests by token budget only to improve throughput. Each request's score depe
server.py's batcher thread groups pending requests by token budget only to improve throughput. Each request's score depends only on its own tokens. There is no cache, memo or cross-request statistic, and the threshold is a fixed config value.
✓actual inference uses reviewed weightsserve.sh runs server.py, which loads ./model, the shipped model/ directory. Its config.json, index and safetensors shard
serve.sh runs server.py, which loads ./model, the shipped model/ directory. Its config.json, index and safetensors shards (including score.weight in model-00004) are the reviewed weights. Provenance of those weights: train/run_train2.sh trains run 2 from the run-1 adapter, and train/merge_lora.py merges it into the base as merged2. The trajectory shows merged2 being downloaded and moved into /workspace/output/model, with the threshold set to -0.5 (trajectory#179, trajectory#180).
✓training data provenancearchitecture.json and DATA_MANIFEST.md declare every training source with a pinned revision, and logs/download_manifest.
architecture.json and DATA_MANIFEST.md declare every training source with a pinned revision, and logs/download_manifest.json and logs/download.log record the downloads: - Cartinoe5930/DeepSeek-Prover-V2-generation - Slim205/lean_workbook_RL_V20_total (train split) - Goedel-LM/Lean-workbook-proofs - LukeBailey181 (lean_workbook splits only, via allow_patterns) - deepseek-ai/DeepSeek-Prover-V1 - kfdong/STP_Lean_0320 (shard 0) - DS-Prover-V1.5-RL generations on Goedel-Pset/Lean Workbook statements - mutants of the above - the shipped validation file (half A for training, half B for calibration) miniF2F was fetched from the DeepSeek-Prover-V1.5 minif2f.jsonl only to build decontamination exclusion keys (trajectory#32, data_pipeline/build_pool.py). Peeking at the LukeBailey miniF2F split via the datasets server was inspection only (trajectory#17, trajectory#18). Its download was restricted to the lean_workbook patterns. One undeclared dataset, Slim205/math_res_DeepSeek-Prover-V1.5-SFT_32_100, was downloaded. However, logs/label_stats.txt shows no dsv15sft rows were labeled, and assemble_train.py and make_run2.py draw only from labeled ids of the declared sources. So it never entered training.
✓non specialist initializationarchitecture.json declares the base Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. train/setup_train_box.sh downloa
architecture.json declares the base Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. train/setup_train_box.sh downloads that model to $HOME/base. train/run_train.sh and train/train_cls.py add a fresh 1-logit score head with LoRA, and run 2 warm-starts from the participant's own run-1 adapter. Goedel-Prover-V2 is a theorem prover, not a specialist Lean-validity classifier, and the checkpoint structure (Qwen3 backbone plus a new score head) is consistent with this.
✓harness credentials used only by the harnessSearching the trajectory found no use of ANTHROPIC or OpenRouter keys and no invocation of a judge API. The participant
Searching the trajectory found no use of ANTHROPIC or OpenRouter keys and no invocation of a judge API. The participant used only the Spin compute/storage CLI and HF access for downloads, both of which are allowed. The MLE_JUDGE_API_KEY mention appears only in a grep of the validator CLI's argument definitions (trajectory#138); it was not used.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 5m into the runtook 6m 7s
startup 12.5 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 12.5 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 149.2 s plus 0.2 s for the first request, 23.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed2h 41m into the runtook 5m 15s
startup 12.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 12.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 131.6 s plus 0.2 s for the first request, 26.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed2h 47m into the runtook 5m 7s
startup 12.6 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 12.6 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 84.3 s plus 0.2 s for the first request, 41.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
All rented machines are terminated; total spend is $8.87 of $10. Before handing in, a last read of the final server.py: