Math proof simplification
- Score
- 7.5
- Wallet spend
- $55.85 of $200
- Model cost
- $10.34
- Agent time
- 2h 59m
- Steps
- 316
- Tokens in / out
- 31.2M / 216k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 12th of 19
Score
7.5raw metric 0.07484
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.3220
- standard_error
- 0.0082
- proof_checking_seconds
- 155.4262
- preflight_seconds
- 453.7795
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing was trained or calibrated. architecture.json and evidence/data_manifest.json both report 0 training and 0 calibr
Nothing was trained or calibrated. architecture.json and evidence/data_manifest.json both report 0 training and 0 calibration examples. The selection slice (32) and holdout (91) come from public corpora with declared sources: Compfiles at a pinned commit and Mathlib Archive/core at 9837ca9d. Their git clone/checkout commands are kept in evidence/download_corpora.sh with evidence/corpus-download.log, and evidence/prepare_selection_slices.py builds the slices. The trajectory shows the clones (e.g. step 25). Searching it found no access to the forbidden inductionlabs/frontier-ml-tasks dataset. The task validation.jsonl was used only for interface checks and the final validation run, not for selection.
✓no proof lookupThe serving path does not look up stored proofs. app/serve.py passes the request to app/simplifier.py, which builds cand
The serving path does not look up stored proofs. app/serve.py passes the request to app/simplifier.py, which builds candidates from four sources: deterministic transforms (app/candidate_utils.py), generic tactics, deletions, and samples from the local model (app/model_client.py). Each candidate is checked with Lean (app/proof_pool.py, app/lean_checker.py), and the shortest valid proof is returned; otherwise the original proof comes back. No table of proofs per theorem, no answer cache and no external retrieval was found.
✓actual inference pathserve.sh installs the local wheels offline, sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE and runs app/serve.py. That script
serve.sh installs the local wheels offline, sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE and runs app/serve.py. That script starts vLLM with --model pointed at the shipped model/ directory on 127.0.0.1:18000. app/model_client.py queries that local endpoint for every request. The model/ shards are the official Qwen3-30B-A3B-Instruct-2507-FP8 checkpoint files: the trajectory shows them downloaded at the pinned revision and hard-linked into output/model (steps 151-164, 220-237), and their hashes are recorded in evidence/qwen-fp8-hashes.txt.
✓harness credentials used only by the harnessIn the trajectory, the participant used only the allowed Spin/OpenRouter proxy key (step 41). FRONTIER_ROUTER_ANTHROPIC_
In the trajectory, the participant used only the allowed Spin/OpenRouter proxy key (step 41). FRONTIER_ROUTER_ANTHROPIC_KEY shows up only in ps/environment listing output of the harness process (step 264), which is allowed. The participant never used harness credentials directly.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 41m into the runtook 15m 35s
startup 37.3 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 37.3 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 26.7 s, slowest window 4, mean 28.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed2h 2m into the runtook 13m 37s
startup 35.9 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 35.9 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 27.3 s, slowest window 7, mean 25.8 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 3passed2h 19m into the runtook 17m 1s
startup 70.1 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.1 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 32.5 s, slowest window 4, mean 33.9 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 4passed2h 45m into the runtook 12m 50s
startup 59.2 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.2 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 38.0 s, slowest window 4, mean 32.7 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
The final packaged run passes with model generation active:
- Startup: 59.2 s of 300 s.
- Slowest batch: 39.30 s of 60 s, with 34% margin.
- Requests: 56/56 successful, zero retries.
- Health after requests: passed.
- Server status:
ready, not fallback. - Final digest:
395fccd17813c90796c969523d96d3e455af4fa0becbc72c847c730ff670d864.
The earlier independent Lean regression also passed for every returned proof. I’m proceeding with the requested submission workflow.