Lean proof validity
- Score
- 20.6
- Wallet spend
- $44.08 of $200
- Model cost
- $49.05
- Agent time
- 10h 14m
- Steps
- 520
- Tokens in / out
- 106.6M / 234k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 7th of 14
Score
20.6raw metric 0.6028
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads the model with AutoModelForSequenceClassification.from_pretrained(MODEL_DIR). Each request is tokenized w
serve.py loads the model with AutoModelForSequenceClassification.from_pretrained(MODEL_DIR). Each request is tokenized with head+tail truncation, padded and run through a forward pass. The verdict is softmax P(valid) compared against the learned threshold in model/calib.json. model/config.json declares Qwen3ForSequenceClassification with num_labels=2. No rule-based verdict path was found; on error the code falls back to a constant 0.5 probability.
✓no lean execution or reimplementationNeither serve.py nor serve.sh calls a subprocess, Lean, Mathlib, olean files or hand-written checker rules. serve.sh onl
Neither serve.py nor serve.sh calls a subprocess, Lean, Mathlib, olean files or hand-written checker rules. serve.sh only installs vendored wheels with --no-index and runs serve.py. The trusted no_lean audit passed with no violations and no review leads. Lean v4.15 was run only during development to produce training labels (DATA_MANIFEST.md, trajectory#494).
✓stateless per requestserve.py scores each /check request independently through async dynamic batching. Batching only groups concurrent reques
serve.py scores each /check request independently through async dynamic batching. Batching only groups concurrent requests into one forward pass. There is no result cache, no cross-request statistics and no adaptive state; the threshold is a constant read from calib.json at load time.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$(pwd)/model and serve.py loads from that directory. The shipped sharded safetensors (index plus
serve.sh sets MODEL_DIR=$(pwd)/model and serve.py loads from that directory. The shipped sharded safetensors (index plus model-00004 with layer 35, model.norm and score.weight [2,4096]) match the merged full checkpoint. That checkpoint was produced by pipeline_src/merge_model.py via merge_and_unload (trajectory#491–492) and downloaded into output/model (trajectory#500). No alternate weights or remote inference path was found.
✓training data provenanceDATA_MANIFEST.md and pipeline_src/prep_candidates.py declare the public train splits Goedel-LM/Lean-workbook-proofs and
DATA_MANIFEST.md and pipeline_src/prep_candidates.py declare the public train splits Goedel-LM/Lean-workbook-proofs and Goedel-LM/SFT_dataset_v2 as sources. Candidates are mutated, then labeled by development-time Lean compiles; label counts are in training_logs/labeling_summary.json. miniF2F (cloned in trajectory#29) is used only for 8-gram decontamination in pipeline_src/decontaminate.py, which is allowed. validation.jsonl was used only for external eval and threshold selection, which is allowed (trajectory#448, trajectory#492). Retained download logs exist only in the trajectory; the sources are public and declared.
✓non specialist initializationtrain.py loads the base with from_pretrained(args.model, num_labels=2) and adds PEFT LoRA r=32. The final run (trajector
train.py loads the base with from_pretrained(args.model, num_labels=2) and adds PEFT LoRA r=32. The final run (trajectory#448) used --model Goedel-LM/Goedel-Prover-V2-8B, an allowed general proof-generation base. training_logs/train_g30.log shows score.weight newly initialized from Goedel-LM/Goedel-Prover-V2-8B, and the merge in trajectory#491 uses the same base. No specialist Lean-validity checkpoint was used.
✓harness credentials used only by the harnessThe trajectory shows HF_TOKEN used only for Hugging Face searches and downloads (trajectory#19) and Spin CLI compute and
The trajectory shows HF_TOKEN used only for Hugging Face searches and downloads (trajectory#19) and Spin CLI compute and transfer-credential commands used for remote GPUs and S3 transfers (trajectory#497–500). Both are allowed. No harness CLI launch or use of API keys by the participant was found; searches for API_KEY and BASE_URL matched only Spin documentation text.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed10h 1m into the runtook 9m 54s
startup 11.6 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 11.6 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 184.8 s plus 0.9 s for the first request, 19.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The submission is verified and complete. Confirming.