Lean proof validity
- Score
- 57.5
- Wallet spend
- $168.53 of $200
- Model cost
- $49.47
- Agent time
- 6h 2m
- Steps
- 282
- Tokens in / out
- 63.5M / 199k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
57.5raw metric 0.7876
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads the two members named in inference_config.json (model_goedel2 and model_goedel2_mc) with AutoModelForSeq
server.py loads the two members named in inference_config.json (model_goedel2 and model_goedel2_mc) with AutoModelForSequenceClassification on cuda. Each request is tokenized and run through both models. The softmax probability at each member's valid_index is averaged with weights and compared against a threshold of 0.45. The verdict therefore comes from learned forward passes.
✓no lean execution or reimplementationserver.py and serve.sh only tokenize the text (header stripping and middle truncation to 2048) and run model inference.
server.py and serve.sh only tokenize the text (header stripping and middle truncation to 2048) and run model inference. There is no subprocess, Lean/REPL call or hand-written Lean checking logic. The trusted no_lean audit passed with no violations and no review leads. Lean was used only to label training data offline during development.
✓stateless per requestserver.py has no result cache, memo table or cross-request persistent state. The per-member queue loop batches concurren
server.py has no result cache, memo table or cross-request persistent state. The per-member queue loop batches concurrent requests for throughput only. Each output depends only on that request's own encoded text.
✓actual inference uses reviewed weightsserve.sh runs server.py with HF offline mode. server.py loads the local directories model_goedel2 and model_goedel2_mc f
serve.sh runs server.py with HF offline mode. server.py loads the local directories model_goedel2 and model_goedel2_mc from inference_config.json. Both have safetensors shards and index files in the weight inventory. The trajectory shows these directories were copied from the trained checkpoints ck/v3_goedel2 and ck/v3_goedel2_mc.
✓training data provenancearchitecture.json and DATA_MANIFEST.md declare every corpus. Statements come from public Lean Workbook and Goedel-Pset-v
architecture.json and DATA_MANIFEST.md declare every corpus. Statements come from public Lean Workbook and Goedel-Pset-v1, plus public Cartinoe5930/DeepSeek-Prover-V2-generation attempts, all downloaded by training/gen/download_data.py (log retained). Proof attempts were self-generated by open provers via training/gen/gen.py, and mutants were made by make_mutants.py; all were labeled with Lean during development. The validation file was used only for selection and calibration. miniF2F was used only for decontamination filtering in decontam.py and build_statements.py. Lean-workbook-proofs and NuminaMath-LEAN were downloaded but not used by the shipped pipeline. The trajectory shows train_v3 was built only from the declared sources (170600 examples).
✓non specialist initializationarchitecture.json declares the base as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision, and setup_train.sh downloads
architecture.json declares the base as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision, and setup_train.sh downloads it by snapshot_download. The train args JSONs and train log show both members fine-tuned from /root/models/Goedel-Prover-V2-8B, with LoRA merged and the score head newly initialized. This means the head was never a pretrained validity classifier. The trajectory confirms the trained run started from this base.
✓harness credentials used only by the harnessSearches of the shipped code and trajectory for harness API keys, base URLs, openrouter and claude -p turned up no use b
Searches of the shipped code and trajectory for harness API keys, base URLs, openrouter and claude -p turned up no use by the participant. The only hits are docs or help text. External tooling was limited to spin compute and storage commands (e.g. sync.py and mutate_new.py).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 45m into the runtook 9m 30s
startup 18.6 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 18.6 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 239.6 s plus 0.3 s for the first request, 14.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed4h 16m into the runtook 7m 41s
startup 21.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 21.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 180.6 s plus 0.2 s for the first request, 19.6 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed5h 55m into the runtook 7m 27s
startup 16.9 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 16.9 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 183.0 s plus 0.2 s for the first request, 19.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Now the architecture.json: