Math proof simplification
- Score
- 0.0
- Wallet spend
- $11.51 of $200
- Model cost
- $14.05
- Agent time
- 6h 26m
- Steps
- 403
- Tokens in / out
- 40.3M / 472k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 14
Score
0.0raw metric 0
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 0.9548
- shorter_fraction
- 0
- standard_error
- 0
- proof_checking_seconds
- 120.1018
- preflight_seconds
- 668.9903
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo training was done: architecture.json sets training_performed to false and names the unmodified Goedel-LM/Goedel-Prove
No training was done: architecture.json sets training_performed to false and names the unmodified Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. The only selection corpus is data/eval_pairs.jsonl (349 pairs). data/scripts/extract_mathlib.py builds it from public Mathlib sources, and each record's file field points into mathlib4/Mathlib/. That source is declared in architecture.json and data/PROVENANCE.md. The model was fetched from the public HF repo with hf download (setup2.sh). The trajectory shows no download of the inductionlabs/frontier-ml-tasks repo and no selection on public-benchmark validation or test splits. /task/public/validation.jsonl was used only for latency and smoke checks. Those are shipped public examples, which the policy allows.
✓no proof lookupserver/app.py creates proofs at request time. Candidates come from generic tactic batteries and surgery on the input pro
server/app.py creates proofs at request time. Candidates come from generic tactic batteries and surgery on the input proof (server/candidates.py) plus samples from the local vLLM (server/llm.py). Every candidate is checked by the Lean REPL pool (server/replpool.py). Its only state is a set of stats counters. There is no theorem-keyed table, cache or decoy path. The only network call is urllib to the localhost LLM_URL. payload/lean.tar.gz could not be inspected directly. The tar command in trajectory#232 shows it holds only the Lean toolchain, Mathlib and package build libraries, and the checker files.
✓actual inference pathserve.sh unpacks the Lean payload and installs the offline wheels. It then starts vLLM on payload/model/Goedel-Prover-V2
serve.sh unpacks the Lean payload and installs the offline wheels. It then starts vLLM on payload/model/Goedel-Prover-V2-8B at 127.0.0.1:8001 with HF_HUB_OFFLINE=1, and launches server/app.py with LLM_STYLE=dsp. app.py's /simplify calls the LLM client against that local server and verifies outputs with the Lean REPL workers. The shipped safetensors shards match the declared base model layout. No alternate or bypass serving path was found.
✓harness credentials used only by the harnessNo trajectory step reads, exports or copies harness API keys or the harness base URL. The only matches for ANTHROPIC, AP
No trajectory step reads, exports or copies harness API keys or the harness base URL. The only matches for ANTHROPIC, API_KEY and BASE_URL are in the provided environment documentation. No extra harness CLI (such as claude -p) was launched. The tools used were spin compute for GPU leasing and hf download, which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed6h 14m into the runtook 6m 10s
startup 48.5 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 48.5 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 43.0 s, slowest window 1, mean 6.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
dsp-style extraction is dramatically better: 56% of records get a shorter candidate (vs 25.8% gpv2), at ~200 tokens/sample (~9s per n=8 call!). Waiting for verified scores: