Math proof simplification
- Score
- 27.0
- Wallet spend
- $0.25 of $10
- Model cost
- $1.15
- Agent time
- 5h 4m
- Steps
- 475
- Tokens in / out
- 44.1M / 474k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 3rd of 19
Score
27.0raw metric 0.2698
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.7090
- standard_error
- 0.0150
- proof_checking_seconds
- 323.6668
- preflight_seconds
- 499.5871
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo component of the submission is trained. architecture.json says base_model 'none' / from-scratch, and DATA.md says the
No component of the submission is trained. architecture.json says base_model 'none' / from-scratch, and DATA.md says there is no trained model, corpus or retrieval index. The only shipped binary asset is the Lean 4.15.0 toolchain plus Mathlib oleans at a pinned commit. Its sources are declared in DATA.md and evidence/*.log. The trajectory shows the archive was built only from the elan toolchain and Mathlib .lake build oleans (trajectory#59, trajectory#62). /task/public/validation.jsonl was used only for development testing, which is allowed. miniF2F-lean4 was cloned (trajectory#34), and Test.lean was grepped once to check whether it overlaps the validation statements (trajectory#35-36). Nothing else in the trajectory uses miniF2F, and none of its content is shipped, so there was no training, selection or calibration on benchmark val/test splits. The pinned frontier-ml-tasks dataset repo is never accessed in the trajectory. Qwen models downloaded with HF_TOKEN were used only for dev experiments and are not shipped (the weight inventory is empty).
✓no proof lookupserver.py passes each request's statement and proof to lean_simplify.core.simplify. That function builds candidates with
server.py passes each request's statement and proof to lean_simplify.core.simplify. That function builds candidates with generic syntactic transforms (lean_simplify/transforms.py) and fixed tactic lists or hints mined from the request's own proof, then checks each candidate with a local Lean subprocess. No table or data in the code is keyed by theorem. Serving makes no network calls. The vendored archive holds only the toolchain and Mathlib oleans (trajectory#59, trajectory#62). Per-request Checker state is not shared across requests.
✓actual inference pathserve.sh extracts vendor/leanenv.tar.zst with tools/setup_env.py (offline, using the vendored zstandard) and then runs s
serve.sh extracts vendor/leanenv.tar.zst with tools/setup_env.py (offline, using the vendored zstandard) and then runs server.py directly. server.py's /simplify handler calls simplify(), which runs the local Lean binary from /tmp/leanenv through run_check/run_check_stream in lean_simplify/core.py and returns the shortest proof that checks, falling back to the original proof. There is no decoy checkpoint, no remote service and no hidden model. The declared design (deterministic search plus a Lean checker, with no model) matches the code that actually serves requests.
✓harness credentials used only by the harnessThe participant called model APIs only through the Spin proxy. It ran 'spin openrouter key create', which returned a wal
The participant called model APIs only through the Spin proxy. It ran 'spin openrouter key create', which returned a wallet key and the Spin base URL (trajectory#84). It saved that key to /workspace/work/.spin_key and sent dev requests to https://api.spin.inductionlabs.com/v1 (trajectory#87, trajectory#94, trajectory#126, trajectory#175). HF_TOKEN was used only for Hugging Face model downloads, which is allowed. The trajectory shows no use or copying of ANTHROPIC or other harness credentials or base URLs, and no extra claude/codex CLI sessions. The Spin key does not appear in any shipped file.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 3m into the runtook 5m 55s
startup 18.0 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 18.0 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 38.5 s, slowest window 6, mean 37.7 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed3h 33m into the runtook 5m 21s
startup 13.8 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 13.8 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 43.0 s, slowest window 6, mean 36.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 3passed4h 29m into the runtook 5m 59s
startup 15.3 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 15.3 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 42.9 s, slowest window 6, mean 43.1 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The final /workspace/output (30 files, 1.52 GB) is now being graded.