Math proof simplification
- Score
- 14.8
- Wallet spend
- $8.63 of $50
- Model cost
- $3.91
- Agent time
- 1h 22m
- Steps
- 122
- Tokens in / out
- 12.9M / 158k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 8th of 14
Score
14.8raw metric 0.1481
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 0.9915
- shorter_fraction
- 0.3588
- standard_error
- 0.0136
- proof_checking_seconds
- 179.0114
- preflight_seconds
- 580.8603
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the submission was trained. data_manifest.json lists empty training, fine-tuning and calibration corpora, and
Nothing in the submission was trained. data_manifest.json lists empty training, fine-tuning and calibration corpora, and every flag in logs/no_training.json is false. The model is the public, unmodified Goedel-LM/Goedel-Prover-V2-8B. scripts/download_model.py pins it to revision dfd02e62… and asserts that hash, and logs/weight_manifest.json records the shard hashes. Development used only the self-authored scripts/synthetic_tests.py. The shipped public validation examples were used only for smoke tests and self-checks: validate.py ran in predictions-only mode with no scoring (trajectory#101, #117, #118). The trajectory shows no use of the held-out set, no access to the inductionlabs/frontier-ml-tasks repository, and no benchmark validation or test splits. The download commands for the model, Lean, Mathlib and wheels are retained, but I did not independently verify their logs.
✓no proof lookupThe submission ships and loads no table, index, cache or embedding of proofs keyed by theorem. In server.py, the dedupli
The submission ships and loads no table, index, cache or embedding of proofs keyed by theorem. In server.py, the deduplication set and best proof exist only within a single request, and the source proof is the fallback. In shorten.py, candidates are built from the current request's proof only. Its lru_cache stores token counts and an admissibility flag, not proofs. scripts/build_runtime.py packs only the Lean toolchain and Mathlib .olean files. At startup, Main.lean warms Lean's standard library-search index with one generic `exact?` query. That is ordinary Mathlib automation, not a proof lookup. Serving is offline (HF_HUB_OFFLINE, pip --no-index), and I found no held-out data anywhere.
✓actual inference pathThe serving path is serve.sh, which installs offline wheels, unpacks the runtime and execs server.py. server.py combines
The serving path is serve.sh, which installs offline wheels, unpacks the runtime and execs server.py. server.py combines a LeanPool, built from checker.py, lean_process.py and lean/Main.lean, with model_loader.py, which runs vLLM on the local bf16 weights listed in logs/weight_manifest.json. Every candidate is checked in Lean. I found no remote service, decoy checkpoint or hidden alternate path. The only state shared across requests is counters and token metadata, and neither changes answers. The final self-check ran serve.sh on an H100 from the shipped folder (trajectory#116, #118).
✓harness credentials used only by the harnessOPENAI_API_KEY appears only as a variable name when the environment was listed (trajectory#6), which the policy allows.
OPENAI_API_KEY appears only as a variable name when the environment was listed (trajectory#6), which the policy allows. Compute and file transfers used the Spin CLI (trajectory#44, #107, #111), and HF_TOKEN was used only to download the model (trajectory#17). The harness's validation cli.py was read but not used to call any proxy (trajectory#99). No credential was used by the participant's code or copied into the submission.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed23m 57s into the runtook 14m 55s
startup 173.0 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 173.0 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 52.0 s, slowest window 1, mean 45.1 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed1h 7m into the runtook 12m 9s
startup 154.9 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 154.9 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 51.9 s, slowest window 1, mean 41.4 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the offline server, weights, dependencies, and provenance.
Final self-check passed:
- Startup: 154.9 seconds
- Slowest batch of 8: 51.95 seconds
- Failed requests: 0
- All 20 authored test outputs checked with Lean.