Lean proof validity
- Score
- 28.6
- Wallet spend
- $8.29 of $10
- Model cost
- $58.83
- Agent time
- 4h 4m
- Steps
- 82
- Tokens in / out
- 15.0M / 159k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 5th of 14
Score
28.6raw metric 0.6428
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads the model from ./model with AutoModelForCausalLM and runs a forward pass on each request. It takes the s
server.py loads the model from ./model with AutoModelForCausalLM and runs a forward pass on each request. It takes the softmax of the ' No' and ' Yes' logits at the end of a fixed prompt and sets valid = p > threshold, where the threshold is 0.5 in inference_config.json. The verdict comes from the model's output, not from rules or heuristics.
✓no lean execution or reimplementationNeither serve.sh nor server.py calls Lean, a REPL, a subprocess or a rule-based checker. The trusted no_lean audit passe
Neither serve.sh nor server.py calls Lean, a REPL, a subprocess or a rule-based checker. The trusted no_lean audit passed with no violations, review leads or uninspected files. Lean/Mathlib v4.15.0 and the REPL (training/download_data.sh, the compile worker) were used only during development to label training data, which the policy allows.
✓stateless per requestIn server.py each request is encoded on its own with a fixed prompt prefix and suffix, and truncation applies only to th
In server.py each request is encoded on its own with a fixed prompt prefix and suffix, and truncation applies only to that request. The batcher thread groups concurrent requests only for throughput, using left padding and an attention mask. There is no cache, no stored history and no state across requests that could change a verdict.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE=1 and runs server.py, which loads ./model, the shipped model/*.safetensors shards indexed b
serve.sh sets HF_HUB_OFFLINE=1 and runs server.py, which loads ./model, the shipped model/*.safetensors shards indexed by model/model.safetensors.index.json. The provenance chain is: training/gpu_train.sh trains a LoRA adapter, training/merge_and_export.py merges it into the base and saves it, and training/finalize.sh copies the result into the output's model directory. No other model or remote endpoint is used when serving.
✓training data provenancedata_manifest.json, architecture.json and training/download_data.sh declare the sources: the train splits of public HF c
data_manifest.json, architecture.json and training/download_data.sh declare the sources: the train splits of public HF corpora (Goedel-LM/Lean-workbook-proofs, AI-MO/NuminaMath-LEAN with amc_aime/math_test/math_train sources excluded, internlm/Lean-Workbook), labels from the candidate's own Lean compiles, and the training half of the allowed /task/public/validation.jsonl. The miniF2F informal statements were cloned and used only to build an exclusion list for decontamination (training/prep_sources.py overlaps_minif2f; trajectory#14, trajectory#15). No miniF2F test/valid statements or attempts were used as training data. One minor inconsistency: data_manifest.json describes validation as held out, while build_dataset.py/architecture.json show its training half was used. The policy allows either use.
✓non specialist initializationThe base model is Goedel-LM/Goedel-Prover-V2-8B, a general Lean prover model and not a validity classifier, pinned to re
The base model is Goedel-LM/Goedel-Prover-V2-8B, a general Lean prover model and not a validity classifier, pinned to revision dfd02e6271a58375dfbf3ece0175277cf6b6a89a. training/dl_base.sh and training/gpu_setup.sh download it from HF. training/train.py loads it with --base, adds LoRA, and trains on the Yes/No logits, and training/merge_and_export.py merges the result. The checkpoint layout matches a merged Qwen3-8B-architecture model.
✓harness credentials used only by the harnessThe trajectory shows only HF_TOKEN used for HF dataset and model downloads, plus spin CLI compute and storage use; both
The trajectory shows only HF_TOKEN used for HF dataset and model downloads, plus spin CLI compute and storage use; both are allowed. The environment listing masked secrets (trajectory#3). Searches found no use of Anthropic or other harness API keys or base URLs by the participant's code or shipped files.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 57m into the runtook 6m 34s
startup 14.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 14.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 132.1 s plus 0.1 s for the first request, 26.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
All leases are stopped, every evidence file listed in architecture.json exists, and the output folder is unchanged since the passing selfcheck. Submitting now.