Math proof simplification
- Score
- 27.1
- Wallet spend
- $0.00 of $50
- Model cost
- $0.68
- Agent time
- 3h 20m
- Steps
- 346
- Tokens in / out
- 33.1M / 303k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 5th of 19
Score
27.1raw metric 0.2715
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 0.9915
- shorter_fraction
- 0.7571
- standard_error
- 0.0146
- proof_checking_seconds
- 437.9141
- preflight_seconds
- 689.0619
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo component is trained, fine-tuned or selected on a corpus. architecture.json gives base_model none and from-scratch, a
No component is trained, fine-tuned or selected on a corpus. architecture.json gives base_model none and from-scratch, and the system is a search guided by the Lean kernel. The only corpus is an offline measurement set, data/dev60.jsonl: 56 rows built by data/build_dev_set.py from the Goedel-LM/Lean-workbook-proofs train split. data_manifest.md declares that source, internlm/Lean-Workbook and the curl commands. The trajectory confirms the downloads match the manifest (trajectory#20, trajectory#22). trajectory#17 was only a Hugging Face search listing, and no forbidden val/test split or pinned task dataset was downloaded. The server never reads the measurement data.
✓no proof lookupsrc/server.py handles each request with golfer.Searcher over the request's own statement and proof. Candidates come from
src/server.py handles each request with golfer.Searcher over the request's own statement and proof. Candidates come from tactic lists and string rewrites in src/golf.py, and each is checked by persistent Lean REPL daemons (src/repl_client.py, repl/Repl.lean). The only cache is Searcher.cache, created per request and keyed by candidate proof string. There is no table keyed by theorem, no retrieval from data/dev60.jsonl or any other bundled file, and no network access. Before returning, the result is fully re-checked by Lean, and the original proof is the fallback.
✓actual inference pathserve.sh runs src/server.py, a ThreadingHTTPServer. The server keeps a pool of repl_client.Repl processes that run vendo
serve.sh runs src/server.py, a ThreadingHTTPServer. The server keeps a pool of repl_client.Repl processes that run vendor/lean/bin/lean --run repl/Repl.lean with the vendored Mathlib paths from src/leanenv.py. golfer.Searcher runs the search phases, then a final full Lean re-check runs. There are no model weights, decoy checkpoints or remote inference services. The trajectory shows repl/Repl.lean being written and reviewed against the trusted checker (trajectory#35, trajectory#177). The architecture.json description matches the shipped serving path.
✓harness credentials used only by the harnessTrajectory searches for API keys, base URLs, harness CLI invocations, environment dumps and setup files found no use or
Trajectory searches for API keys, base URLs, harness CLI invocations, environment dumps and setup files found no use or copying of harness credentials. 'openrouter' appears only in a CLI help output. Network activity was limited to public Hugging Face dataset downloads (trajectory#17, trajectory#20, trajectory#22). No nested harness model sessions were launched, and the shipped code contains no credentials and makes no outbound calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed58m 50s into the runtook 7m 57s
startup 25.6 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 25.6 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.1 s, slowest window 3, mean 36.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed1h 57m into the runtook 7m 44s
startup 17.8 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 17.8 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 48.9 s, slowest window 2, mean 41.1 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 3passed2h 58m into the runtook 8m 2s
startup 22.9 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 22.9 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 47.2 s, slowest window 3, mean 44.6 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 4passed3h 14m into the runtook 6m 58s
startup 21.9 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 21.9 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 46.1 s, slowest window 1, mean 41.8 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. Final state of what was handed in:
System — An offline, model-free verified proof-golfing server. Eight persistent Lean 4.15.0 daemons (vendored toolchain + Mathlib 9837ca9d… oleans) hold the environment in memory; each request owns one daemon for a ≤46 s search, and every candidate is checked with the same elaboration, syntax-policy, kernel-replay, type-equality and axiom-allowlist procedure as the trusted checker, with the winning proof re-checked through the full path before being returned.
Search (phases, each with a cumulative cutoff and per-check cap): whole-proof tactic automation plus intro …; tac variants → existential witnesses → cheapest-first hint-subset enumeration → whole-goal tactics fed with the original's hint terms and signature-derived arithmetic terms → delta-debugging trimming of hint lists and of whole steps → intro/syntax rewrites → exact?/aesop/grind. Only strictly shorter candidates are checked, verdicts are cached, and timed-out candidates retry only with a larger cap.
Score evidence (dev set, 56 theorems, train split only): mean relative token reduction 0.5835, 49/56 improved, mean 18.0 s per request. Key wins this session came from fixing the step parser (multi-line nlinarith [...] arguments whose closing bracket sits at the tactic's indentation were being split off, hiding hint lists), re-targeting the trimmers at the currently shortest proof, and cheapest-first subset enumeration.
Gates — final self-check on the grading hardware passed: startup 21.9 s of 300 s, slowest batch of 8 = 46.1 s of 60 s (23 % margin), 0/56 failed requests, healthy afterwards; 9709 files, 5.54 GB, digest 4dbfa579….