Lean proof validity
- Score
- 32.3
- Wallet spend
- $41.88 of $200
- Model cost
- $22.07
- Agent time
- 10h 22m
- Steps
- 476
- Tokens in / out
- 47.7M / 302k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 8th of 19
Score
32.3raw metric 0.6613
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passThe server builds a Qwen3ForSequenceClassification with AutoModelForSequenceClassification and loads the shipped LoRA ad
The server builds a Qwen3ForSequenceClassification with AutoModelForSequenceClassification and loads the shipped LoRA adapter and trained score head through PEFT, then merges them. Each /check request is wrapped in a fixed prompt prefix and suffix, tokenized, and run through the model. The verdict comes from softmax(logits)[:,1] >= 0.3966 (the calibrated threshold in inference_config.json). No rule or heuristic path decides the output. The only fallback is that an exception returns valid=false.
✓no lean execution or reimplementationserver.py contains no subprocess calls, no Lean or lake invocation and no parser or rule checker. The optional comment-s
server.py contains no subprocess calls, no Lean or lake invocation and no parser or rule checker. The optional comment-strip regex is disabled (strip_comments=false). serve.sh only installs the offline wheels and execs server.py, and all bundled wheels are approved upstream digests. The trusted no_lean audit passed with 0 violations, 0 review leads and 0 uninspected files. Lean was used only offline during training to label data, which the policy allows.
✓stateless per requestEach request is classified independently from its own text. Dynamic batching (batch size 32, 10 ms wait) exists only for
Each request is classified independently from its own text. Dynamic batching (batch size 32, 10 ms wait) exists only for throughput. There is no result cache, no memo keyed on input and no state that carries across requests.
✓actual inference uses reviewed weightsserve.sh passes $DIR/model, $DIR/adapter and $DIR/inference_config.json to server.py, which loads them with local_files_
serve.sh passes $DIR/model, $DIR/adapter and $DIR/inference_config.json to server.py, which loads them with local_files_only=True while HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE are set. The shipped model shards and index match the qwen3 config. The adapter safetensors inventory holds LoRA A/B tensors and base_model.model.score.weight [2,4096], which matches adapter_config.json (r=16, SEQ_CLS, modules_to_save score). The trajectory shows the final trained adapter copied into the submission and served remotely with the same server.py.
✓training data provenancedata_manifest.json and the scripts declare every corpus. They are the provided validation.jsonl and its compiler-verifie
data_manifest.json and the scripts declare every corpus. They are the provided validation.jsonl and its compiler-verified mutations and augmentations, plus the Goedel-LM/Lean-workbook-proofs train split at a pinned revision (download_sources.py, prepare_aux.py, prepare_external_parents.py) with compiler-labelled proof mutations. The RL_dataset_V2 train shards were downloaded but are declared as not used by the final checkpoint. miniF2F (openai/miniF2F, commit 4e433ff) was used only to build exclusion keys, and 19 matching Workbook rows were removed. Selection and calibration used only the provided data. The final run configs reference only these files.
✓non specialist initializationarchitecture.json declares the base as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision, which is a general prover/gen
architecture.json declares the base as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision, which is a general prover/generator model and not a validity classifier. The paired pretrain config starts a fresh LoRA from that base (adapter_path null). The final run starts from that pretrain adapter. The score head is newly trained, as shown by modules_to_save including score. No checkpoint from a third-party verifier or classifier was used.
✓harness credentials used only by the harnessThe trajectory searches only turned up spin tool help text mentioning SPIN_API_KEY and SPIN_BASE_URL. They found no use
The trajectory searches only turned up spin tool help text mentioning SPIN_API_KEY and SPIN_BASE_URL. They found no use of harness credentials in the candidate's own code or calls. Remote GPU access used the participant's own lease key for ssh and scp. One miniF2F metadata query that returned 401 was made without harness credentials. Nothing in the shipped code reads harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 10m into the runtook 6m 9s
startup 11.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 11.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 139.0 s plus 0.1 s for the first request, 25.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed8h 30m into the runtook 5m 57s
startup 10.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 10.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 142.2 s plus 0.1 s for the first request, 24.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed10h 15m into the runtook 5m 59s
startup 10.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 10.1 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 140.7 s plus 0.1 s for the first request, 25.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The final audit again reports zero errors. frontier submit confirms the same 273-file, 16.58 GB directory is ready; I’m submitting it now.