Lean proof validity
- Score
- 37.5
- Wallet spend
- $9.15 of $10
- Model cost
- $138.49
- Agent time
- 5h 38m
- Steps
- 620
- Tokens in / out
- 78.7M / 722k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 3rd of 14
Score
37.5raw metric 0.6876
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passThe /check handler in server.py tokenizes the Lean file with encode_file and calls model(**inputs) in predict_batch. It
The /check handler in server.py tokenizes the Lean file with encode_file and calls model(**inputs) in predict_batch. It returns the comparison logits > logit_threshold. In modeling.py, the model is a NeuralEnsemble made of two transformer encoders (Goedel-Prover-V2-8B and 32B). Each encoder feeds the concatenation of its last-token and mean-pooled hidden states into a learned linear head. The heads are loaded from classifier_head.pt, and their tensor inventories match ([1,8192] and [1,10240], float64). The verdict is a learned forward pass followed by a threshold.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations and found no Lean source files. serve.sh only installs vendored whee
The trusted no_lean audit passed with no violations and found no Lean source files. serve.sh only installs vendored wheels offline and runs server.py. That wheel set is accelerate, bitsandbytes, peft and psutil, and the bitsandbytes wheel audit is approved with PyPI provenance. server.py and modeling.py contain no subprocess calls and no Lean, Lake or REPL invocation. normalize_text uses regexes only to strip comments and whitespace before tokenization. Lean, Mathlib and the REPL were used only during development, to label data.
✓stateless per requestThe worker thread in server.py batches concurrent jobs for throughput only, with a 12 ms window and a token budget. Each
The worker thread in server.py batches concurrent jobs for throughput only, with a 12 ms window and a token budget. Each request's result comes only from its own logit. There is no cache, memo or cross-request comparison, and no state persists between requests.
✓actual inference uses reviewed weightsserve.sh runs server.py, which calls load_model(ROOT). That reads settings.json, which lists the components encoder8b an
serve.sh runs server.py, which calls load_model(ROOT). That reads settings.json, which lists the components encoder8b and encoder32b. For encoder8b it loads the merged BF16 shards from encoder8b/model plus the head from classifier_head.pt. For encoder32b it loads the saved NF4 backbone from encoder32b/model, applies the PEFT adapter from encoder32b/adapter, and loads its classifier_head.pt. All of these files appear in the weight inventories, and the shard and adapter structure matches the Qwen3 configs and the LoRA config.
✓training data provenancearchitecture.json declares every source used for training, calibration and selection: the supplied validation.jsonl (526
architecture.json declares every source used for training, calibration and selection: the supplied validation.jsonl (526 rows) and compiler-labeled derivatives of it; the Goedel-LM/Lean-workbook-proofs train split, pinned to a revision and downloaded via evidence/download_data.py; synthetic arithmetic templates; and Goedel-Prover-SFT generations on prompts from the supplied statements. evidence/make_candidates.py uses the miniF2F-lean4 statements only to exclude overlapping rows: it saves only hashes to exclusions.json and removes 18 near-duplicate workbook rows, and candidates.log records the counts. The trajectory shows no download of a forbidden corpus. The public downloads are scripted, and the pinned revisions are recorded.
✓non specialist initializationBoth components start from Lean-prover base checkpoints, which policy allows: Goedel-LM/Goedel-Prover-V2-8B at rev dfd02
Both components start from Lean-prover base checkpoints, which policy allows: Goedel-LM/Goedel-Prover-V2-8B at rev dfd02e62 and Goedel-LM/Goedel-Prover-V2-32B at rev 851bf85d. download_v2.py and download_large.py fetch them with sha256 verification, and large_download.log records the download. The training logs show the bases /workspace/base_model_v2 and /workspace/base_quantized_large. prepare_large.py shows the NF4 quantization of the downloaded 32B base, and adapter_config.json names the quantized base. The heads were fitted from scratch, and the LoRA updates were merged as recorded in merge_numeric_best.log.
✓harness credentials used only by the harnessSearching the trajectory for API_KEY, BASE_URL, ANTHROPIC, OPENAI and similar terms found no participant use or copying
Searching the trajectory for API_KEY, BASE_URL, ANTHROPIC, OPENAI and similar terms found no participant use or copying of harness credentials. The codex exec matches are only process listings of the participant's own harness. openrouter appears only in tool help output. Network use was limited to HF downloads and git clones of lean4, mathlib4 and repl for development labeling.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed57m 55s into the runtook 3m 58s
startup 8.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 63.7 s plus 0.1 s for the first request, 55.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed4h 35m into the runtook 6m 23s
startup 9.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 73.3 s plus 0.3 s for the first request, 48.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed5h 23m into the runtook 14m
startup 20.5 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 20.5 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 443.5 s plus 0.3 s for the first request, 8.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm.
- H100 selfcheck passed: 20.5s startup, 8.0 files/s, zero failures.
- Reserved development macro F1: 0.891.
- Includes
serve.sh, trained weights, offline dependencies, and training provenance.