Math proof simplification
- Score
- 27.3
- Wallet spend
- $0.00 of $200
- Model cost
- $4.69
- Agent time
- 11h 31m
- Steps
- 1,933
- Tokens in / out
- 189.2M / 1.8M
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 4th of 19
Score
27.3raw metric 0.273
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.7797
- standard_error
- 0.0140
- proof_checking_seconds
- 429.0794
- preflight_seconds
- 671.6530
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in this submission is trained. architecture.json declares base_model "none", training.performed false and empty
Nothing in this submission is trained. architecture.json declares base_model "none", training.performed false and empty corpus lists. It says /task/public was used only for validate.py and for comparing configurations. The development scripts scripts/experiment.py and scripts/timing.py default to /task/public/validation.jsonl, which is a shipped public example file and is allowed. The trajectory reads only those public examples (e.g. trajectory#93). It shows no download from the pinned dataset repository, no use of its private folder and no test split: searches for frontier-ml-tasks, test.jsonl and /private found nothing. The shipped assets are the Lean v4.15.0 toolchain and the Mathlib build at the pinned commit. Their download and copy commands are kept in scripts/download_toolchain.sh, scripts/download_mathlib.sh and scripts/copy_assets.sh, with logs in logs/.
✓no proof lookupEvery candidate is generated at request time from that request's statement and proof. simplify/candidates.py uses generi
Every candidate is generated at request time from that request's statement and proof. simplify/candidates.py uses generic tactic lists, inlines facts and lemma names taken from the request's own proof, and builds cleaned or ablated variants, combos and hill-climbing steps. Its docstring says no table of proofs is stored and nothing is keyed by the theorem. Searches for embedded proof maps, dataset files, hashing of theorems and network calls found nothing. simplify/server.py keeps all search state local to each request and has no cross-request cache. Each candidate is accepted only after Lean checks it. If none passes, the server falls back to the original proof.
✓actual inference pathserve.sh sets LEAN_BIN, CHECK_DIR and LEAN_PATH (pointing at the shipped toolchain, mathlib build and checker), then run
serve.sh sets LEAN_BIN, CHECK_DIR and LEAN_PATH (pointing at the shipped toolchain, mathlib build and checker), then runs exec python3 simplify/server.py. The server's POST /simplify handler calls _search and _search_core. These generate candidates and check them through the CheckPool of persistent lean --server workers (simplify/lean_pool.py, toolchain/bin/lean, checker/Check.lean). _finalize then confirms the chosen proof on FINAL_POOL using the official frontier_check procedure. There are no model weights, no decoy model and no external service on the serving path.
✓harness credentials used only by the harnessI searched the trajectory for ANTHROPIC, _API_KEY, BASE_URL, openrouter, hf_hub and "claude -p". The only matches were s
I searched the trajectory for ANTHROPIC, _API_KEY, BASE_URL, openrouter, hf_hub and "claude -p". The only matches were spin documentation about SPIN_API_KEY, curl -sI connectivity checks against huggingface.co, and an environment process listing that shows a claude-router process (around trajectory#775), which is allowed. No step uses or copies harness credentials or base URLs, and no step launches extra harness CLIs. The shipped code makes no network or API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed10h 24m into the runtook 8m 16s
startup 0.8 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs ag
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 0.8 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.9 s, slowest window 5, mean 47.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed11h 21m into the runtook 7m 57s
startup 1.0 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs ag
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 1.0 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.9 s, slowest window 4, mean 46.8 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final state of /workspace/output — a training-free Lean 4 proof simplifier:
serve.sh→POST /simplify(stdlib-only Python HTTP service, no GPU, no network, no wheels to install)simplify/candidates.py— 9 candidate families (one-token tactics, hypothesis-applied tactics, inlinedhavefacts, mechanical shortenings, block ablation,<;> (try …)dedup, greedy fact-list thinning, the 2-square Cauchy form, two-step/shape tactics)checker/Check.lean— the officialfrontier_checkplus a batchedfrontier_check_many_fastvariant; every returned proof is accepted by the reference checker (0 invalid of 47 verified)toolchain/(Lean v4.15.0) +mathlib/(9837ca9d…, compiled oleans),architecture.json,README.md,scripts/,logs/
Verification before handing in:
| gate | result |
|---|---|
frontier selfcheck afd600cf0a8d | PASSED — startup 1.0 s / 300 s, slowest batch of 8 48.22 s / 60 s (20 % margin), 0 % failed requests / 5 % allowed, healthy after requests |
| score vs. best config | v20 = 0.29118, final adds validation-0023 (23 → 20 tokens, +0.130 on that item); all other items byte-identical to v20 on spot check (0022 158, 0027 144, 0019 294) |
| tree hygiene | 0 symlinks, 0 __pycache__/*.pyc, architecture.json valid JSON, serve.sh executable |
Submitted at 2026-10-21 07:04:35 UTC, with ~30 min of the 12 h budget to spare.