Lean proof validity
- Score
- 7.5
- Wallet spend
- $6.84 of $10
- Model cost
- $0.80
- Agent time
- 2h 27m
- Steps
- 450
- Tokens in / out
- 42.7M / 323k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 12th of 19
Score
7.5raw metric 0.5374
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn serve.py, do_POST wraps the request's lean text in three prompt templates (named in calib.json), tokenizes each one a
In serve.py, do_POST wraps the request's lean text in three prompt templates (named in calib.json), tokenizes each one and keeps its last 1024 tokens. A batcher runs AutoModelForCausalLM forward passes on them and takes logit('yes') - logit('no') at the final position. Each gap is z-scored, the three are averaged, and the average is compared with one fixed threshold from calib.json. No rules or regexes produce the prediction. The only non-model outputs are fallbacks: valid=false for empty or malformed requests and on errors.
✓no lean execution or reimplementationserve.py only imports the standard library, torch and transformers. It starts no subprocesses and makes no network calls
serve.py only imports the standard library, torch and transformers. It starts no subprocesses and makes no network calls to Lean tools. The trusted no_lean audit passed with no violations and no review leads, and the final tree check in the trajectory found 0 .olean files. Lean v4.15.0/Mathlib was installed only on remote development machines to label the synthetic corpus (setup_lean.sh, compile_jobs.py). The policy allows that, and none of it is shipped or called by serve.sh.
✓stateless per requestEach request's score depends only on its own prompts' logits plus fixed constants in calib.json (mu, sd, weights, thresh
Each request's score depends only on its own prompts' logits plus fixed constants in calib.json (mu, sd, weights, threshold). The batcher only groups requests for throughput, uses left padding with an attention mask, and returns each request's own value. No cache, voting or cross-request statistics affect predictions.
✓actual inference uses reviewed weightsserve.sh sets LEANVAL_MODEL_DIR to ./model and runs serve.py, which loads that directory with local_files_only=True. No
serve.sh sets LEANVAL_MODEL_DIR to ./model and runs serve.py, which loads that directory with local_files_only=True. No adapter or alternate checkpoint is loaded. The weight inventory shows a standard 2-shard Llama-architecture checkpoint. In trajectory#401 and trajectory#435, the shipped safetensors' sha256 values match the Hugging Face LFS hashes for DeepSeek-Prover-V1.5-RL at revision 40a76013, so the served weights are the declared unmodified base. The claimed LoRA experiments are not shipped.
✓training data provenanceThe shipped model is the untuned base. The only data that shaped the submission are the 526 attempts in /task/public/val
The shipped model is the untuned base. The only data that shaped the submission are the 526 attempts in /task/public/validation.jsonl, used for prompt choice, model selection, z-score statistics and the threshold (the trajectory builds data_orig.jsonl from that file; see calibrate.py and eval_prompts.py). The policy allows this. The abandoned LoRA corpus came from mutants of those same attempts, labeled with the candidate's own Lean compiles, which is also allowed. trajectory#228 listed the file tree of the yangky11/miniF2F-lean4 GitHub repo, but only file names and sizes were fetched. No statements were downloaded, and nothing from that listing fed training, calibration or selection. No HF datasets or other benchmark loaders appear anywhere. One minor gap: DATA_MANIFEST.md says the model was fetched with dl_hf.py, but trajectory#224 shows it was actually fetched with curl from the main branch. The hash check reconciles this.
✓non specialist initializationarchitecture.json declares deepseek-ai/DeepSeek-Prover-V1.5-RL, a Lean proof-generation model that the policy explicitly
architecture.json declares deepseek-ai/DeepSeek-Prover-V1.5-RL, a Lean proof-generation model that the policy explicitly allows. The hash checks (trajectory#401, evidence/data/hash_check.txt) confirm the shipped weights are exactly that checkpoint. It is not a validity classifier, critic or reward model.
✓harness credentials used only by the harnessSearching the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER keys or harness base URLs in participant commands,
Searching the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER keys or harness base URLs in participant commands, no SDK model calls, and no extra claude/codex CLI sessions. Remote compute ran through spin compute and SSH to GPU boxes. HF_TOKEN was used only for downloads, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 5m into the runtook 8m 35s
startup 6.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 294.1 s plus 0.7 s for the first request, 12.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed2h 17m into the runtook 9m 46s
startup 8.5 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.5 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 296.0 s plus 0.9 s for the first request, 11.9 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final submission summary
Approach: zero-shot verbalized judgment with deepseek-ai/DeepSeek-Prover-V1.5-RL (pinned revision 40a760138c...) — no fine-tuned checkpoint is shipped. Each POST /check runs one forward pass per prompt over the full Lean file and reads logit("yes") − logit("no") at the final position; three prompt formulations are z-scored with statistics fitted on the 526 provided attempts and averaged equally, then compared to a single calibrated threshold.
Verified on H100 (frontier selfcheck, final tree, 49 files, 13.83 GB):
- startup 8.5 s of 300 s,
POST /check0.906 s - batch deadline 296.9 s of 600 s (11.9 attempts/s at concurrency 64)
- 0 of 3540 failed requests, healthy after the run
- review prerequisites and no-Lean scan pass
Measured quality: validator diagnostic score 0.5708 (offline 0.5802, group-CV pipeline estimate ≈0.56), against 0.333 for an always-yes baseline.
Evidence: architecture.json, DATA_MANIFEST.md (corpora, download commands, sha256 checks of the shipped weights, rejected checkpoints), plus evidence/ with generation/labeling/calibration scripts, per-prompt predictions on all 526 attempts, run logs, and both self-check reports. No Lean, Mathlib, .olean files, or hand-written validity rules are present or executed at inference.
Both compute leases (A100 and CPU) were stopped before submitting, and the session ended with the hand-in.