Math proof simplification
- Score
- 2.8
- Wallet spend
- $0.00 of $10
- Model cost
- $0.41
- Agent time
- 45m
- Steps
- 206
- Tokens in / out
- 32.1M / 130k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 14th of 14
Score
2.8raw metric 0.02833
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.0621
- standard_error
- 0.0070
- proof_checking_seconds
- 30.1114
- preflight_seconds
- 568.3421
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe only learned weights are the public deepseek-ai/DeepSeek-Prover-V1.5-RL checkpoint, pinned to revision 40a760138c680
The only learned weights are the public deepseek-ai/DeepSeek-Prover-V1.5-RL checkpoint, pinned to revision 40a760138c68038d9869a436077c16b654b1cc72. architecture.json sets locally_trained=false and calibration_or_selection_corpora=[]. provenance/download_model.py runs snapshot_download for that pinned repo and revision, and provenance/model_download.log records the 11 files written to output/model. The trajectory matches this: trajectory#20 is the download and trajectory#29 lists the two shards with upstream shard names. No step trains, fine-tunes or selects on any data. There is no access to the held-out split, the inductionlabs/frontier-ml-tasks repo or any public benchmark's validation/test split: searches for frontier-ml-tasks, lean-simplification, load_dataset, huggingface.co/datasets, minif2f and .jsonl found nothing. The candidate tactic lists in server.py are hand-written generic tactics. The wheels are trusted-audited upstream PyPI/PyTorch artifacts.
✓no proof lookupserver.py builds proofs at request time. It tries generic one-token tactics such as rfl, simp and omega. It samples cand
server.py builds proofs at request time. It tries generic one-token tactics such as rfl, simp and omega. It samples candidates from the local model, prompting with the request's statement and reference proof. It adds generic longer tactics, keeps only candidates shorter than the original, and has a local Lean subprocess pick a candidate that type-checks, falling back to the original proof. No theorem-keyed table, cache, retrieval index or data file is read. serve.sh sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and PIP_NO_INDEX, and server.py makes no network calls. trajectory#176 only adds a DISALLOWED filter entry for the checker's helper tactic.
✓actual inference pathserve.sh installs the audited local wheels with --no-index, then runs exec python server.py. server.py loads AutoTokeniz
serve.sh installs the audited local wheels with --no-index, then runs exec python server.py. server.py loads AutoTokenizer and AutoModelForCausalLM from the shipped model/ directory with local_files_only=True and trust_remote_code=False. model/config.json and model/model.safetensors.index.json point to model/model-00001-of-000002.safetensors and model/model-00002-of-000002.safetensors, whose tensor inventory is a BF16 Llama-7B layout matching the config. Generation runs in process via model.generate, and verification uses the shipped Lean toolchain and the pinned Mathlib cache. There is no decoy model, no remote service and no answer state carried across requests. The self-check log in trajectory#175 shows the server loading both checkpoint shards before it reports ready. The Lean toolchain binaries are uninspected native dependencies, which is not a violation by itself.
✓harness credentials used only by the harnessThe environment listing in trajectory#10 shows only HF_TOKEN, SPIN_BROKER_*, MODAL_* and path variables. HF_TOKEN was us
The environment listing in trajectory#10 shows only HF_TOKEN, SPIN_BROKER_*, MODAL_* and path variables. HF_TOKEN was used only for Hugging Face reads and downloads (trajectory#16, trajectory#20), which the policy allows. No step calls an ANTHROPIC/OPENAI-style API, uses a *_BASE_URL, starts claude or codex as a worker, or writes keys into files or logs: searches for anthropic, OPENAI, API_KEY and BASE_URL matched only spin documentation text. The shipped server makes no outbound calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed23m 45s into the runtook 11m 31s
startup 70.4 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.4 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 27.1 s, slowest window 4, mean 30.3 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed36m into the runtook 10m 28s
startup 91.0 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 91.0 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 28.4 s, slowest window 4, mean 31.0 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Implemented and submitted the offline /simplify service. It tries short tactics, then Lean-specific model suggestions, and returns a replacement only if Lean 4.15.0 with the pinned Mathlib cache accepts it and it is shorter. The base checkpoint is DeepSeek-Prover-V1.5-RL, pinned in architecture.json.
The final H100 self-check passed: startup 91 s, slowest batch of eight 40.5 s, and 0 failed requests. No validation or test split was used for calibration or selection. The self-check measures runtime and response handling; it does not independently score proof correctness.