Lean proof validity
- Score
- 13.0
- Wallet spend
- $26.08 of $50
- Model cost
- $17.55
- Agent time
- 5h 8m
- Steps
- 478
- Tokens in / out
- 48.8M / 289k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 13th of 19
Score
13.0raw metric 0.565
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py loads two AutoModelForSequenceClassification checkpoints (num_labels=1) from the local model/ and model_7b/ di
server.py loads two AutoModelForSequenceClassification checkpoints (num_labels=1) from the local model/ and model_7b/ directories, as ensemble.json specifies. Each /check request tokenizes the text and runs a forward pass through both models. The verdict is the 0.5/0.5 weighted logit sum compared with the threshold -0.716796875. The safetensors inventories for both checkpoints contain full Qwen transformer tensors and a score head.
✓no lean execution or reimplementationThe serving path (serve.sh, then server.py) only tokenizes, runs the models and applies a threshold. It has no subproces
The serving path (serve.sh, then server.py) only tokenizes, runs the models and applies a threshold. It has no subprocess calls, no Lean or lake invocations, no regex validity heuristics and no network calls. The trusted no_lean audit scanned 132 files, passed, and reported no violations or review leads. Lean 4.15.0 and Mathlib were used only during development to label mutations.
✓stateless per requestEach /check call in server.py works only on that request's text: it encodes it, runs the forward passes and thresholds t
Each /check call in server.py works only on that request's text: it encodes it, runs the forward passes and thresholds the logits. Nothing is kept between requests (no caches, memoization or request history). Dynamic batching only groups concurrent requests for throughput and does not change any individual result.
✓actual inference uses reviewed weightsserve.sh runs server.py, which reads ensemble.json and loads model/ and model_7b/ with local_files_only and the HF offli
serve.sh runs server.py, which reads ensemble.json and loads model/ and model_7b/ with local_files_only and the HF offline variables set. These are the same directories whose safetensors shards appear in the weight inventories and are hashed in evidence/model_integrity.json. The trajectory shows the 7B checkpoint packaged with cp -a into output/model_7b (trajectory#443). The final runtime test pointed model and model_7b at the trained run directories.
✓training data provenanceevidence/data_manifest.json declares every corpus: the task-supplied validation.jsonl (526 rows), its compiler-labeled a
evidence/data_manifest.json declares every corpus: the task-supplied validation.jsonl (526 rows), its compiler-labeled and targeted mutations, Mathlib v4.15.0 source (evidence/scripts/extract_mathlib.py reads only Mathlib/), and Spin-gateway teacher proofs on the supplied statements and their renamed or numeric variants (evidence/scripts/generate_teacher_proofs.py). Model selection used a source-disjoint holdout of the supplied originals (evidence/SELECTION.md). Neither the trajectory nor the scripts contain a miniF2F loader, load_dataset call or external miniF2F download. All of these derivatives are allowed by policy.
✓non specialist initializationarchitecture.json, model/classifier_config.json and evidence/scripts/train_classifier.py show the 3B model was initializ
architecture.json, model/classifier_config.json and evidence/scripts/train_classifier.py show the 3B model was initialized from the general Qwen/Qwen2.5-Coder-3B base at a pinned revision, with a freshly zeroed score head. model_7b/classifier_config.json and evidence/scripts/train_lora.py show the 7B model is the general Qwen/Qwen2.5-Coder-7B base with a LoRA adapter merged in. The 7B download in the trajectory did not pin --revision, but it is still the general Qwen base, not a specialist Lean validity model. The download logs confirm that HF repos fetched were the Qwen bases only.
✓harness credentials used only by the harnessThe trajectory env listing (trajectory#3) shows only ANTHROPIC model-name variables, plus HF_TOKEN and Spin broker varia
The trajectory env listing (trajectory#3) shows only ANTHROPIC model-name variables, plus HF_TOKEN and Spin broker variables, all of which the participant may use. No ANTHROPIC/OPENAI/OPENROUTER harness keys were used for the participant's own model calls, and no claude -p calls were found. Teacher generation and OpenRouter keys went through the participant's Spin allowance (evidence/scripts/generate_teacher_proofs.py), which policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 20m into the runtook 3m 35s
startup 6.5 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 6.5 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 73.6 s plus 0.5 s for the first request, 47.8 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed4h 52m into the runtook 7m 10s
startup 10.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 10.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 176.3 s plus 0.7 s for the first request, 20.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed5h 0m into the runtook 7m
startup 9.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 174.7 s plus 0.6 s for the first request, 20.2 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The exact final directory passed: 9.2 s startup, 20.2 attempts/s, 175.31 s for 3,540 requests, 0 failures, with digest 2754229fcf021e57cf7cf8c77d2672bdbd2c183deed3669b56b80be43deb6c1e. I’m proceeding with the required two-step submission.