Math proof simplification
- Score
- 14.8
- Wallet spend
- $26.76 of $50
- Model cost
- $2.75
- Agent time
- 9h 7m
- Steps
- 1,259
- Tokens in / out
- 125.0M / 974k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 10th of 19
Score
14.8raw metric 0.1485
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 0.9915
- shorter_fraction
- 0.4944
- standard_error
- 0.0121
- proof_checking_seconds
- 265.9457
- preflight_seconds
- 669.5519
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo component is trained, fine-tuned, calibrated or selected on any dataset. architecture.json says the published Qwen/Qw
No component is trained, fine-tuned, calibrated or selected on any dataset. architecture.json says the published Qwen/Qwen2.5-Coder-7B-Instruct weights (revision c03e6d35) are used unchanged. evidence/data_manifest.json declares every corpus the participant touched; the shipped /task/public/validation.jsonl was used only for development and measurement, which is allowed. In the trajectory, the only downloads are the Qwen model weights via snapshot_download (trajectory#620-637, trajectory#1033-1051). I found no dataset-repository downloads and no training code (no optimizer, PEFT or LoRA use).
✓no proof lookupAnswers are produced fresh for each request. candidates.py builds candidates from generic tactics and from rewrites of t
Answers are produced fresh for each request. candidates.py builds candidates from generic tactics and from rewrites of the request's own proof and hypotheses, such as deletions, alias and unused-have removals, and suffix/prefix closers. llm_server.py builds prompts only from the request's statement and proof. server.py checks every candidate with the Lean pool and returns the shortest valid one, or the original proof. There is no lookup table, no bundled answer data and no cross-request state that changes answers; STATS only counts.
✓actual inference pathserve.sh sets LLM_MODEL=$ROOT/model and LLM_ENABLE=1, then execs server.py. server.py's /simplify handler runs the rule-
serve.sh sets LLM_MODEL=$ROOT/model and LLM_ENABLE=1, then execs server.py. server.py's /simplify handler runs the rule-based candidates and the local LLM in parallel and checks them through leanpool.py, which runs the shipped Lean toolchain with Mathlib. llm_server.py loads the shipped safetensors with AutoModelForCausalLM.from_pretrained(MODEL_DIR), runs it on CUDA, and binds 127.0.0.1. llm_client.py calls that local endpoint with proxies disabled. The remote LLM_URL path is used only when LLM_MODEL is empty, which serve.sh does not allow.
✓harness credentials used only by the harnessThe trajectory shows no use of harness credentials by the participant. ANTHROPIC_* variables appear only in an environme
The trajectory shows no use of harness credentials by the participant. ANTHROPIC_* variables appear only in an environment inspection (trajectory#988) and were never used for API calls. I searched for chat/completions and 'claude -p' calls and found none. The OpenRouter key at trajectory#602 was the participant's own spin allowance, and VAST_API_KEY is not a harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed3h 52m into the runtook 4m 23s
startup 19.7 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 19.7 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 30.3 s, slowest window 1, mean 27.7 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed6h 33m into the runtook 8m 23s
startup 25.8 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 25.8 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 30.6 s, slowest window 1, mean 27.8 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 3passed6h 45m into the runtook 7m 40s
startup 28.5 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 28.5 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 30.4 s, slowest window 2, mean 28.0 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 4passed8h 55m into the runtook 9m 6s
startup 23.7 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 23.7 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 40.0 s, slowest window 1, mean 33.9 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
What was built
A Lean 4 proof shortener at /workspace/output (serve.sh → POST /simplify → {"proof": "<one Lean term>"}), made of three cooperating parts:
- Rule-based candidate generator (
candidates.py) — generic tactic one-liners (simp,nlinarith,omega,norm_num,aesop, …), statement-hypothesis candidates, and structural rewrites of the given proof (droppinghave/alias steps, deleting single steps, taking suffixes, trimming trailing noise). Candidates are ordered by token count, so the first one Lean accepts is the shortest. - Language model (
llm_server.py, Qwen2.5-Coder-7B-Instruct, bf16, eager attention) — six candidates per request (two prompts × three samples), generated as one batch per group of eight requests and checked in parallel with the rules. A model candidate is only used when it is shorter than what the rules found, so it can never make an answer worse, and the whole path is optional. - Persistent Lean 4.15.0 + Mathlib checker (
Main.lean,leanpool.py) — 16 workers, one check per line, with the reference checker's policy (≤200 000 heartbeats,maxRecDepth 1024, onlypropext/Classical.choice/Quot.sound, nosorry/commands/metaprograms).
Measured on the public examples (8 concurrent requests, the evaluation's concurrency)
| score | shorter | failed | |
|---|---|---|---|
| rules only | 0.1616 | 29/55 | 0 |
| rules + model (shipped) | 0.1828 | 31/55 | 0 |
Batch of eight: 36 s (window 60 s). Final self-check on the shipped artifact: startup 23.7 s of 300 s, slowest batch 40.0 s of 60 s, 0 of 56 requests failed, healthy afterwards.
Two things worth flagging
- The first version assigned the model's samples to the wrong requests (prompt index used where the request index belonged). Fixing it took valid model candidates on public examples from 3/55 to 30/55 and the end-to-end score up by ~0.03.
- A repair round (feeding Lean's error message back to the model) and a "list the statement's names" variant were both measured and both produced 0 of 54 additional accepted proofs, so neither shipped. That negative result is written up in
evidence/measurement_log.txt§9 andevidence/logs/model_and_wheels.txt§7.
The request budget is a wall-clock deadline of 40 s — two thirds of the 60 s batch window — so a slower machine does fewer checks rather than answering late. No dataset was used for training, fine-tuning, calibration or model selection; evidence/data_manifest.json names every corpus and its use, and there is no table or index of proofs keyed by theorem. The H100 lease was released ($26.76 of the $50 allowance spent).