Math proof simplification
- Score
- 23.7
- Wallet spend
- $0.87 of $10
- Model cost
- $17.24
- Agent time
- 2h 50m
- Steps
- 365
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 4th of 19
Score
23.7raw metric 0.2368
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.8672
- standard_error
- 0.0140
- proof_checking_seconds
- 575.4587
- preflight_seconds
- 664.4681
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json declares the base model, Goedel-LM/Goedel-Prover-SFT at pinned revision 5b03a13d. It also declares thr
architecture.json declares the base model, Goedel-LM/Goedel-Prover-SFT at pinned revision 5b03a13d. It also declares three train-split public corpora, each at a pinned revision. download_and_prepare_data.py pulls exactly those files with hf_hub_download at the pinned revisions and keeps only DeepSeek rows with split=='train'. It also drops benchmark-prefixed items and any statement that appears in /task/public/validation.jsonl, then forms 80 dev and 220 train items from train-split data. data_manifest.json and data_prep.log record this. build_sft_dataset.py builds the simplified targets with the local Lean checker. train_simplifier.py validates on the dev split derived from train data. The trajectory shows the base model downloaded with snapshot_download at the declared revision (trajectory#228) and LoRA training run on a Spin A100 (trajectory#205, trajectory#228). The adapter was copied back via Spin (trajectory#280). No undeclared corpus and no use of validation or test material for training was found.
✓no proof lookupserver.py makes candidates by running the merged model, using a simplifier prompt and a direct prompt. It then applies o
server.py makes candidates by running the merged model, using a simplifier prompt and a direct prompt. It then applies one-token finishers, ProofSimplifier tactic search and golfing checked by Lean, and a final safety check that falls back to the original proof. It has no lookup tables, answer files or network calls. In proof_simplifier.py the only cache is checked_cache, which lives for one call and is keyed by proof string. The shipped lean_runtime.tar.zst holds only the Lean toolchain, the Mathlib .oleans and the checker sources (trajectory#181). The sigCache in FastChecker caches statement signatures, not stored proofs (trajectory#30). No mapping from statement to proof was found.
✓actual inference pathserve.sh installs the vendored wheels, extracts the Lean runtime and execs server.py with the shipped base_model and lor
serve.sh installs the vendored wheels, extracts the Lean runtime and execs server.py with the shipped base_model and lora_adapter, with HF_HUB_OFFLINE set. server.py loads base_model, applies the shipped adapter with PeftModel and merge_and_unload, and calls model.generate on the GPU for the /simplify requests. Every output either comes from this model or is that output refined by Lean, so the declared weights are actually used when serving.
✓harness credentials used only by the harnessThe trajectory (antigravity-cli harness) shows no use by the participant of the harness's model-provider credentials: no
The trajectory (antigravity-cli harness) shows no use by the participant of the harness's model-provider credentials: no ANTHROPIC or GEMINI keys and no frontier-ml-tasks access. Its external compute went through the permitted Spin CLI (trajectory#205, trajectory#228, trajectory#280). The other network calls were localhost curl and the uv installer on the Spin VM (trajectory#224).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 13m into the runtook 7m 4s
startup 77.6 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.6 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 16.4 s, slowest window 4, mean 21.4 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed2h 40m into the runtook 7m 21s
startup 76.1 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 76.1 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 19.1 s, slowest window 4, mean 22.3 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests