Math proof simplification
- Score
- 4.6
- Wallet spend
- $0.01 of $10
- Model cost
- $10.28
- Agent time
- 53m
- Steps
- 754
- Tokens in / out
- 92.3M / 237k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 13th of 14
Score
4.6raw metric 0.04581
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.0876
- standard_error
- 0.0089
- proof_checking_seconds
- 41.6159
- preflight_seconds
- 588.3482
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in this submission is trained. architecture.json declares base_model "none" / "from-scratch", lists no training
Nothing in this submission is trained. architecture.json declares base_model "none" / "from-scratch", lists no training evidence files, and calls the system a rule-based, syntax-guided simplifier. The shipped code has no model, weights or training loop. The only learned-free 'selection' was hand-tuning the candidate heuristics, which the session did on /task/public/validation.jsonl (trajectory#57 onward, e.g. the batch evaluation at trajectory#536). That file is one of the shipped public examples, which the policy allows. The session shows no download of the pinned inductionlabs/frontier-ml-tasks repository and no use of a public benchmark's validation or test split. The bundled archives are a Lean toolchain fetched from the GitHub release and a tar of locally built Mathlib oleans (trajectory#391-392), not proof corpora.
✓no proof lookupIn server.py, do_POST calls simplifier.generate_candidates(statement, proof). That function builds candidates only from
In server.py, do_POST calls simplifier.generate_candidates(statement, proof). That function builds candidates only from the request itself: fixed one-line tactics (decide, omega, ring, norm_num and so on), dropping arguments from the last simp/linarith/norm_num call, keeping only the last tactic block, and pruning unused `have` lines. LeanVerifier.verify_candidates_batch then checks each candidate by running the local Lean binary as a subprocess. No table, cache, index or embedding store maps theorems to proofs, and the code makes no network calls. The bundled files are a Lean toolchain archive, a Mathlib olean archive and two checker oleans. The session shows these oleans were compiled from verifier source that only checks a proof's type and axioms (trajectory#361-364, trajectory#159), and the session shows where every bundled file came from (trajectory#540).
✓actual inference pathserve.sh installs the vendored zstandard wheel (covered by a trusted audit), unpacks Lean 4.15.0 and the Mathlib oleans
serve.sh installs the vendored zstandard wheel (covered by a trusted audit), unpacks Lean 4.15.0 and the Mathlib oleans into /tmp, copies FastCheck.olean and ProofCheck.olean onto LEAN_PATH, and then runs server.py. server.py builds a LeanVerifier and handles /simplify with generate_candidates plus one batched `lean --json` call. The first candidate Lean accepts is returned only if it is shorter; otherwise the original proof is returned. Nothing is carried between requests except the fixed verifier configuration, there is no remote service, and no unused checkpoint or alternate path is shipped.
✓harness credentials used only by the harnessThe participant called LLMs only through a Spin OpenRouter-proxy key it created with `spin openrouter key create` (traje
The participant called LLMs only through a Spin OpenRouter-proxy key it created with `spin openrouter key create` (trajectory#119), sending requests to api.spin.inductionlabs.com (trajectory#120, trajectory#286). The policy explicitly allows this. The only mention of OPENROUTER_API_KEY is in the harness's own launch command, which appeared in ps output (trajectory#341). No participant command reads, echoes or copies harness credentials, and no additional harness CLI sessions were launched. HF_TOKEN was used only for a read-only model search (trajectory#57).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed50m 25s into the runtook 2m 47s
startup 15.3 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 15.3 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 12.5 s, slowest window 4, mean 13.4 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Final Submission Complete
The Lean 4 Proof Simplification system has been verified and successfully submitted via frontier submit --confirm.
System Summary & Verification Highlights
-
Submission Bundle (
/workspace/output/):serve.sh: Self-contained offline initialization script. Installs vendoredzstandard, decompresses Lean 4.15.0 and Mathlib oleans to/tmp/, setsLEAN_PATH, and starts the production server.server.py: Threaded HTTP server responding onGET /healthandPOST /simplify. Robust fail-closed architecture returning the valid original proof on unexpected timeouts or failures.simplifier.py: Rule-based and syntax-guided simplification engine executing:- High-value 1-liner decision procedures (
omega,ring,norm_num,linarith,aesop, etc.). - Tactic argument ablation for list-based tactics (
nlinarith [...],simp [...]). - Dead-declaration pruning for unreferenced
havehypotheses. - Fast batched verification via custom precompiled
FastCheck.oleanrunning Lean with-j 1to prevent CPU thrashing.
- High-value 1-liner decision procedures (
architecture.json: Formatted architecture specification (base_model: none,base_revision: from-scratch).packages/: Bundled Lean 4.15.0 binary toolchain, Mathlib oleans archive, and precompiledFastCheck.olean&ProofCheck.olean.wheels/: Vendoredzstandardwheel for offline execution.
-
Self-Check Results (
frontier selfcheck 7d6bd0b386d9):- Status: PASSED
- Startup Time: 15.3s (Limit: 300s, 95% margin).
- Batch Latency: 18.054s slowest window of 8 concurrent requests (Limit: 60s, 70% margin).
- Failed Requests: 0% (56/56 requests answered HTTP 200).
- Post-Request Health: Passed.