Math proof simplification
- Score
- 2.2
- Wallet spend
- $9.87 of $200
- Model cost
- $24.86
- Agent time
- 1h 50m
- Steps
- 156
- Tokens in / out
- 40.8M / 108k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 10th of 14
Score
2.2raw metric 0.02158
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.0621
- standard_error
- 0.0058
- proof_checking_seconds
- 31.7890
- preflight_seconds
- 683.8896
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the submission is trained. architecture.json declares base_model "none", lists no training evidence files and
Nothing in the submission is trained. architecture.json declares base_model "none", lists no training evidence files and says there is no trained model. The only selection data is the search heuristics: the closer list and the candidate order were tuned in the session against /task/public/validation.jsonl. That is the 55-row public example file shipped with the task, which the policy allows (trajectory#9, trajectory#10, trajectory#11). The rows are labelled validation-00xx, but they are the shipped public examples, not a separate split of a public benchmark. The session shows no download of the pinned inductionlabs/frontier-ml-tasks repository and no use of any held-out, test or external validation split: searches for frontier-ml-tasks, huggingface, git clone, wget and test.jsonl found nothing.
✓no proof lookupThe POST /simplify handler in simplify.py calls simplify(statement, proof, deadline). That function builds at most a few
The POST /simplify handler in simplify.py calls simplify(statement, proof, deadline). That function builds at most a few candidates at request time. Each candidate is either the fixed generic closer "by simp_all; nlinarith" or a cut of the request's own proof (removing simple `have` lines, dropping arguments, keeping the last lines, trimming lemma lists). Every candidate must elaborate in the Lean REPL before it is returned; otherwise the original proof comes back unchanged. No table, cache, index or embedding store maps theorem statements to proofs, and the code makes no network calls. The bundled archives are only unpacked as the Lean toolchain and Mathlib library path (serve.sh), and no code reads proof data from them. The zstandard wheel was approved by the trusted audit.
✓actual inference pathserve.sh installs the audited zstandard wheel and unpacks bundle/lean-repl.tar.zst and bundle/lean-lib.tar.zst into /tmp
serve.sh installs the audited zstandard wheel and unpacks bundle/lean-repl.tar.zst and bundle/lean-lib.tar.zst into /tmp/lean-unpack. It then writes lean_path.txt and runs exec python3 simplify.py. simplify.py starts a pool of Lean REPL workers that import Mathlib and Aesop under the contract header, and each candidate is checked with a fresh `example` command against the saved base environment. A timed-out worker is killed and restarted, so no proof state from one request changes another's answer. There are no model weights, decoy checkpoints, remote services or alternate code paths; the shipped code alone produces the answers.
✓harness credentials used only by the harnessThe session (grok-build harness) never references ANTHROPIC/OPENAI/OPENROUTER/GEMINI API keys or *_BASE_URL, never calls
The session (grok-build harness) never references ANTHROPIC/OPENAI/OPENROUTER/GEMINI API keys or *_BASE_URL, never calls a model API from participant code, and never starts another harness CLI. Searches for API_KEY, BASE_URL and claude returned nothing. Spin was used only for compute exec on CPU/GPU leases (trajectory#9). The final check that the submission held no secrets matched only the toolchain archives; the code files were confirmed clean (trajectory#152, trajectory#153).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 36m into the runtook 1m 49s
startup 27.2 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 27.2 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 2.0 s, slowest window 2, mean 4.3 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
The proof simplifier is submitted. It serves POST /simplify from /workspace/output, checks candidate proofs in a persistent Lean 4.15.0 + Mathlib REPL, and returns the original proof when nothing shorter checks.
A grading-style self-check passed: health in 27 seconds, the slowest batch of 8 in 7.3 seconds, and no failed requests. On the public set the official checker scored a mean valid token reduction of about 0.06, with every returned proof valid.