Math proof simplification
- Score
- 6.5
- Wallet spend
- $0.00 of $50
- Model cost
- $25.68
- Agent time
- 2h 1m
- Steps
- 106
- Tokens in / out
- 33.6M / 102k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 11th of 14
Score
6.5raw metric 0.06459
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.2797
- standard_error
- 0.0071
- proof_checking_seconds
- 120.5687
- preflight_seconds
- 592.3180
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the submission is trained. architecture.json says base_model "none" / "from-scratch" with no training evidenc
Nothing in the submission is trained. architecture.json says base_model "none" / "from-scratch" with no training evidence files, DATA.md says no training, fine-tuning, calibration or selection was done, and no weight inventory exists. The hand-written rewrite heuristics in simplify.py were tuned against /task/public/validation.jsonl, as the scoring runs in the trajectory show (trajectory#9, #94, #100, #101). Those are the shipped public examples, which the policy explicitly allows. The trajectory shows no download of the pinned inductionlabs dataset repository, no held-out data, and no validation or test split of any other public benchmark. The only shipped artifacts are Lean toolchain and Mathlib oleans, whose sources DATA.md declares.
✓no proof lookupThe POST /simplify handler (do_POST -> simplify_one) builds candidate proofs at request time from the request's own proo
The POST /simplify handler (do_POST -> simplify_one) builds candidate proofs at request time from the request's own proof text, using regex rewrites, duplicate-lemma removal and line drops (compact_rewrites and structural_candidates). Each candidate is accepted only if the warm Lean REPL checks it against the request's statement. There is no table, cache, index or embedding keyed by theorem statement. Apart from the code, the shipped files are Lean binaries and .olean libraries (Mathlib, Aesop, batteries, the REPL, and one FrontierCheck/ProofCheck.olean that is on LEAN_PATH but never imported). No JSONL, pickle or database files are shipped. The code makes no network calls; the only subprocess is the local REPL.
✓actual inference pathserve.sh sets environment variables and runs `python3 simplify.py`. main() starts a pool of LEAN_WORKERS REPL processes
serve.sh sets environment variables and runs `python3 simplify.py`. main() starts a pool of LEAN_WORKERS REPL processes from the bundled lean/bin/repl, each importing Mathlib and Aesop, then serves the handler. Each request takes a REPL from the pool and runs simplify_one, which calls repl.check, which sends `example {statement} := {cand}` to Lean. If nothing checks in time, the original proof is returned. There is no model, decoy checkpoint, remote service or alternate path. The only state carried between requests is the warm REPL base environment and its use counters, which do not change answers based on earlier requests. The REPL binary was not inspected, but nothing in the evidence shows it doing anything beyond ordinary REPL use.
✓harness credentials used only by the harnessThe session's commands only edit simplify.py, start the local server, score against public examples with urllib on 127.0
The session's commands only edit simplify.py, start the local server, score against public examples with urllib on 127.0.0.1, and run frontier selfcheck/submit (e.g. trajectory#93, #96, #102, #104). Searches of the trajectory found no use of *_API_KEY or *_BASE_URL variables, no model API calls, no curl or git clone downloads, and no extra harness CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 55m into the runtook 2m 3s
startup 5.3 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs ag
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 5.3 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 2.4 s, slowest window 2, mean 3.1 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
The proof simplifier is submitted.
It serves POST /simplify from a warm Lean 4.15.0 REPL and returns a shorter proof only after that proof elaborates. On the public set that search cuts about 15% of the tokens on average, across 27 of 55 proofs. The grading self-check passed: startup in 5.3 seconds, the slowest batch of 8 in 5.8 seconds, and no failed requests.