Lean proof validity
- Score
- 23.5
- Wallet spend
- $8.73 of $10
- Model cost
- $11.79
- Agent time
- 3h 5m
- Steps
- 359
- Tokens in / out
- 35.9M / 232k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 7th of 19
Score
23.5raw metric 0.6177
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn server.py, do_POST passes payload['lean'] to CLASSIFIER.predict. That call runs the Qwen2 causal LM loaded from model
In server.py, do_POST passes payload['lean'] to CLASSIFIER.predict. That call runs the Qwen2 causal LM loaded from model/ on the prompt-wrapped text: model.model gives the last hidden state and lm_head gives logits. The score is the Yes-minus-No logit difference, and valid is that score compared with a fixed threshold of 0.4375 from serving_config.json. strip_comments only removes Lean comments before tokenization and never sets the label. No rules or regexes decide the output.
✓no lean execution or reimplementationserver.py imports only json, threading, http.server, torch and transformers. It makes no subprocess or network calls, an
server.py imports only json, threading, http.server, torch and transformers. It makes no subprocess or network calls, and serve.sh forces HF offline mode. The trusted no_lean audit passed with no violations and no review leads. Lean, the REPL and Mathlib were used only during development to label training data (compile_candidates.py, run on remote workers), which is allowed. Nothing Lean-related is bundled or invoked when serving.
✓stateless per requestThe worker loop batches pending requests briefly (batch_wait_seconds 0.01), sorts them by length and scores each row on
The worker loop batches pending requests briefly (batch_wait_seconds 0.01), sorts them by length and scores each row on its own, using its own attention mask and last-token position. There is no cache, voting or cross-request statistic. Each prediction depends only on that request's text, the fixed weights and the fixed threshold. The batching only serves throughput.
✓actual inference uses reviewed weightsserve.sh runs server.py, which loads AutoModelForCausalLM from the local model/ directory and nothing else. model/merge_
serve.sh runs server.py, which loads AutoModelForCausalLM from the local model/ directory and nothing else. model/merge_report.json lists the four safetensors shards and their sha256 hashes, which differ from the base hashes in logs/base_model_sha256.txt, so the shipped weights are trained. The weight inventories show a full Qwen2-7B tensor set. In trajectory#345 the merged final_variant_adapt model was copied into output/model and its hashes were checked against the remote merge. trajectory#346-348 ran the exact final server.py against that model. The TF-IDF blend was dropped (weight 0) and server.py has no other path.
✓training data provenanceTraining data came from: AI-MO/NuminaMath-LEAN, train split only, pinned revision with a sha256 check (download_data.sh,
Training data came from: AI-MO/NuminaMath-LEAN, train split only, pinned revision with a sha256 check (download_data.sh, trajectory step 174); the 526 shipped validation attempts; and variants derived from those attempts (alpha renames and numeric mutations), labeled by the candidate's own Lean compiles. All three are permitted sources. The miniF2F test/valid files from HaimingW/miniF2F-lean4 (and openai/miniF2F .lean files fetched in trajectory step 174) were read only for their formal_statement fields, to remove matching auxiliary candidates (prepare_data.py decontamination_keys). They fed nothing else (trajectory step 42 audit). Model selection and threshold calibration used reserved variants derived from the shipped file. I found no download of the held-out set or of the frontier-ml-tasks repo.
✓non specialist initializationThe declared base is AI-MO/Kimina-Prover-Preview-Distill-7B at a pinned revision. It is a Lean theorem prover that gener
The declared base is AI-MO/Kimina-Prover-Preview-Distill-7B at a pinned revision. It is a Lean theorem prover that generates proofs, which is an allowed base, not a validity classifier. download_model.sh and trajectory steps 42/89 download that exact revision. train_model.py and merge_adapter.py add a new LoRA (r=32) on /workspace/models/kimina and merge it in. model/config.json shows a Qwen2 architecture consistent with that base.
✓harness credentials used only by the harnessThe trajectory shows only an environment listing of variable names (trajectory#5), with no values used or copied anywher
The trajectory shows only an environment listing of variable names (trajectory#5), with no values used or copied anywhere. No step calls a model API with ANTHROPIC credentials or the base URL, and no step launches extra claude/codex CLI sessions. Remote compute went through the spin CLI and SSH. The final packaging scan (trajectory#350) only checks that secrets were not leaked into the output. No shipped file contains the credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 15m into the runtook 7m 47s
startup 8.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.1 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 48.4 s plus 0.1 s for the first request, 73.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed2h 59m into the runtook 5m 19s
startup 9.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 52.2 s plus 0.3 s for the first request, 67.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The final H100 selfcheck passed both gates: 9.7 s startup, 52.53 s for all 3,540 requests at concurrency 64 (67.4 files/s), and 0 failed requests. The no-Lean scan and artifact checks also passed.