Math proof simplification
- Score
- 5.3
- Wallet spend
- $7.36 of $10
- Model cost
- $35.11
- Agent time
- 2h 20m
- Steps
- 230
- Tokens in / out
- 54.8M / 128k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 12th of 14
Score
5.3raw metric 0.05299
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.0904
- standard_error
- 0.0098
- proof_checking_seconds
- 49.1293
- preflight_seconds
- 658.1887
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the submission is trained. architecture.json declares base_model 'none' / 'from-scratch', and data_manifest.j
Nothing in the submission is trained. architecture.json declares base_model 'none' / 'from-scratch', and data_manifest.json records training, fine-tuning, calibration and selection as 'none'. The trajectory shows the hand-written candidate list and its order in src/search.py were tuned by Lean sweeps over /task/public/validation.jsonl (trajectory#9 and steps 13-33). An earlier experiment generated proofs on those same public examples with Goedel-Prover (trajectory#8, trajectory#80) and was dropped; no model ships. The policy explicitly allows using the shipped /task/public examples. Searches of the session found no download of the pinned inductionlabs/frontier-ml-tasks repository, no load_dataset call or Hugging Face download, and no use of any other benchmark's validation or test split (no miniF2F).
✓no proof lookupThe POST /simplify handler in src/server.py calls search(statement, proof). That builds candidates in src/search.py from
The POST /simplify handler in src/server.py calls search(statement, proof). That builds candidates in src/search.py from generic tactics, the statement's hypothesis names, variables and numerals, and tactic hints taken from the submitted proof itself. Each candidate is checked live by a Lean server through the frontier_try command and the shortest one that passes is kept. No table, index, cache or embedding keyed by theorem exists. The only bundled data are the Lean/Mathlib toolchain archives, the checker olean (compiled in trajectory#75 from the shipped 3316-byte ProofCheckFast.lean) and a vendored zstd extension. The code makes no network calls.
✓actual inference pathserve.sh unpacks the Lean and Mathlib archives with src/unpack.py (using the vendored zstd decompressor), copies the che
serve.sh unpacks the Lean and Mathlib archives with src/unpack.py (using the vendored zstd decompressor), copies the checker and its prebuilt olean, and runs src/server.py. server.py starts a pool of `lean --server` processes, sends each request's candidates from search.candidates as a document edit, reads the FRONTIER_OK/BAD diagnostics, and picks the shortest by src/metric.py. No model weights or decoy checkpoints ship, and there is no remote service. Pool workers keep Mathlib imported, but each request replaces the whole document text, so earlier requests do not change later answers. The wheels/pip branch does nothing because no wheels are shipped.
✓harness credentials used only by the harnessThe trajectory shows only spin compute exec, Spin-issued S3 transfer sessions (/tmp/session.json, /tmp/model_session.jso
The trajectory shows only spin compute exec, Spin-issued S3 transfer sessions (/tmp/session.json, /tmp/model_session.json) and local Lean/Python commands. Searches for API_KEY, BASE_URL, anthropic, openrouter, XAI and printenv found nothing, and no extra harness CLI sessions were launched. The Goedel-Prover generation ran locally on a Spin GPU machine with transformers (trajectory#8), not through any harness model API.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed2h 6m into the runtook 5m 52s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/src/unpack.py", line 30, in <module> un
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/src/unpack.py", line 30, in <module> unpack(archive, dest, strip) File "/submission/src/unpack.py", line 13, in unpack raw = zstd.decompress(archive.read_bytes()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ zstd.Error: Decompression error: length mismatch -> decomp 0 != 131072 [header]
- 2passed2h 15m into the runtook 4m 1s
startup 26.0 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 26.0 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 13.7 s, slowest window 4, mean 21.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
The self-check passed: startup took 26 seconds and the slowest batch of 8 finished in 27 seconds. I’ll submit this version.