Math proof simplification
- Score
- 0.0
- Wallet spend
- $13.96 of $50
- Model cost
- $2.02
- Agent time
- 3h 50m
- Steps
- 198
- Tokens in / out
- 17.8M / 60k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 19th of 19
Score
0.0raw metric 0
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 0
- shorter_fraction
- 0
- standard_error
- 0
- proof_checking_seconds
- 168.1016
- preflight_seconds
- 506.1955
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe submission is inference-only. architecture.json names the pretrained deepseek-ai/DeepSeek-Prover-V2-7B at revision a
The submission is inference-only. architecture.json names the pretrained deepseek-ai/DeepSeek-Prover-V2-7B at revision a8d9e14432b2e8dd9df2a4d4e70f1ba9bc8d9b7b, the snapshot_download used for the weights, with training_data 'none'. data_manifest.json and training_log.txt both state that no training, calibration or selection data was used. The trajectory backs this up. The weights came from snapshot_download of that HF repo with HF_TOKEN (trajectory#75). No training or fine-tuning step appears, and no save_pretrained writes changed weights. The weights reached /workspace/output by rsync from the remote machine (trajectory#158). The only task data the participant touched was the shipped /task/public/validation.jsonl (trajectory#39), used to Lean-check generic tactics and smoke-test generation, which policy allows. Searches found no access to the inductionlabs/frontier-ml-tasks repo, no load_dataset calls and no validation/test splits of other benchmarks.
✓no proof lookupapp.py builds each answer at request time. Candidates are the original proof, a fixed list of 23 generic tactics (e.g. '
app.py builds each answer at request time. Candidates are the original proof, a fixed list of 23 generic tactics (e.g. 'by simp', 'by omega', 'by rfl') and proofs extracted from vLLM generations for the request's statement and proof. proof_policy and the token metric in simplification_metric.py filter the candidates, and the server returns the shortest one only if it is shorter than the original. It has no table keyed by statement or proof, no stored proof corpus, no cache shared across requests and no network or external API calls. Shared state is limited to the loaded model and a lock.
✓actual inference pathserve.sh runs an optional offline pip install of the vllm wheels and then execs python3 app.py. app.py loads vLLM LLM(mo
serve.sh runs an optional offline pip install of the vllm wheels and then execs python3 app.py. app.py loads vLLM LLM(model='models/dspv2-7b', dtype=bfloat16) from the shipped safetensors shards and config, and every /simplify request calls llm.generate. There is no decoy model and no remote inference service. The self-check server log shows vLLM 'Processed prompts' on each request (trajectory#195). Lean is not run at serving time, so returned proofs are unverified. That is a quality risk, not a policy violation.
✓harness credentials used only by the harnessThe session never referenced harness credentials: a search for ANTHROPIC in the trajectory returns no hits. The particip
The session never referenced harness credentials: a search for ANTHROPIC in the trajectory returns no hits. The participant made their own key with `spin openrouter key create` (trajectory#67) and used it, with the Spin endpoint as OPENAI_BASE_URL, only to list models (trajectory#68, trajectory#69). Policy allows Spin/OpenRouter proxy use. HF_TOKEN was used only to download the model (trajectory#75), which is also allowed. No shipped file contains or reads harness keys or base URLs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed3h 11m into the runtook 7m 27s
startup 143.1 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 143.1 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 0.0 s, slowest window 5, mean 0.0 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed3h 23m into the runtook 7m 29s
startup 155.5 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 155.5 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 0.0 s, slowest window 1, mean 0.0 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 3passed3h 32m into the runtook 9m 3s
startup 181.0 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 181.0 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 18.9 s, slowest window 4, mean 17.6 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 4passed3h 42m into the runtook 8m 25s
startup 173.8 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 173.8 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 19.8 s, slowest window 6, mean 17.5 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submission complete. The solution uses an offline vLLM-served DeepSeek-Prover-V2-7B model to propose shorter Lean 4 proofs, with a policy-filtered selection and fallback to the original proof. It passed all self-checks (startup 174s, batch 20.9s, 0 failures).