Lean proof validity
- Score
- 33.1
- Wallet spend
- $72.91 of $200
- Model cost
- $14.25
- Agent time
- 6h 15m
- Steps
- 342
- Tokens in / out
- 43.5M / 440k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 14
Score
33.1raw metric 0.6657
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn server.py, POST /check puts the request's lean text on a queue. classify() runs AutoModelForSequenceClassification fo
In server.py, POST /check puts the request's lean text on a queue. classify() runs AutoModelForSequenceClassification forward passes for three Qwen3 members. Each member gives a softmax compile-class probability. The probabilities are averaged with weights 0.35/0.35/0.30 and compared with a fixed threshold of 0.51 from inference_config.json. text_utils.py only builds the prompt, normalizes theorem names and strips comments in the code view. No regex or rule produces the prediction. The member skipping in classify() is exact interval bounding on the weighted mean, so it gives the same answer as the full ensemble.
✓no lean execution or reimplementationThe serving code (server.py, text_utils.py, serve.sh) makes no subprocess or network calls and no Lean client calls. ser
The serving code (server.py, text_utils.py, serve.sh) makes no subprocess or network calls and no Lean client calls. serve.sh installs only fastapi, uvicorn, pydantic, bitsandbytes, accelerate, peft and their dependencies from the bundled wheels, all of which have trusted binary audits. The trusted no_lean audit passed with no violations, no Lean source files and no review leads. Lean/lake were used only during development to label data (label_data.py, label_repl.py) and are not part of the server.
✓stateless per requestbatch_worker groups queued requests only for throughput. member_probabilities scores each sequence separately: right pad
batch_worker groups queued requests only for throughput. member_probabilities scores each sequence separately: right padding, last-token classification head, and no cross-sample statistics. classify() decides each index from its own weighted probability against the fixed threshold. There is no cache, voting or cross-request state; STATE holds only the models and settings.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py loads every member listed in inference_config.json: models/raw8 and models/code8 (mer
serve.sh runs server.py. server.py loads every member listed in inference_config.json: models/raw8 and models/code8 (merged BF16 safetensors, each with a [2,4096] score.weight), and models/large32 (an NF4 Qwen3 backbone, quantization_config in config.json) with the PEFT adapter at adapters/large32 (LoRA plus a [2,5120] score head). All these weight files appear in the inventories. No alternate path exists, and the unused models were moved out of the output directory (trajectory#269-285 chunk).
✓training data provenancedownload_data.py pins the training sources: the train-split files of uw-math-ai/APRIL (workbook rows only) and Goedel-LM
download_data.py pins the training sources: the train-split files of uw-math-ai/APRIL (workbook rows only) and Goedel-LM/Lean-workbook-proofs. prepare_data.py reads cat-searcher/minif2f-lean4 valid/test only to build the exclusion sets of names and statements, and those rows feed nothing else (audit_exclusions.py, evidence/exclusion_audit.json). The rest of the data is the shipped provided.jsonl, mutations of that file and of workbook rows, randomized arithmetic, numeric counterfactuals, and Goedel-Prover-V2-8B attempts on workbook train statements. All of it was labeled by the candidate's own Lean runs (evidence/data_manifest.json, generate_attempts.py). The trajectory shows no download of held-out data or other benchmark splits.
✓non specialist initializationarchitecture.json declares Goedel-LM/Goedel-Prover-V2-8B and Goedel-Prover-V2-32B, which are proof generators and allowe
architecture.json declares Goedel-LM/Goedel-Prover-V2-8B and Goedel-Prover-V2-32B, which are proof generators and allowed bases. train.py loads raw/base, the base snapshot downloaded at a pinned revision by download_data.py and generate_attempts.py, with a newly initialized score head; the trajectory log shows 'score.weight newly initialized'. The 32B member is the NF4-quantized base plus a LoRA adapter trained during this task. Nothing indicates a validity classifier or reward model was used as the starting point.
✓harness credentials used only by the harnessSearching the trajectory for API_KEY, OPENAI_API_KEY, BASE_URL and nested harness CLI invocations found nothing. The par
Searching the trajectory for API_KEY, OPENAI_API_KEY, BASE_URL and nested harness CLI invocations found nothing. The participant used the spin compute/s5cmd CLI, ssh to rented GPUs and HF downloads. No step shows harness credentials being used or copied.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed39m 38s into the runtook 8m 1s
startup 18.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 18.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 93.2 s plus 0.2 s for the first request, 37.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed5h 2m into the runtook 14m 47s
startup 29.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 29.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 215.0 s plus 0.2 s for the first request, 16.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the trained classifier ensemble, serve.sh, model weights, and training provenance.
Final offline H100 selfcheck passed:
- Startup: 29.4 seconds
- Throughput: 16.4 files/second
- Failed requests: 0 / 3540
- No-Lean scan: passed
Predictions come from the trained models; inference does not run Lean or Mathlib.