Math proof simplification
- Score
- 19.2
- Wallet spend
- $14.62 of $200
- Model cost
- $13.43
- Agent time
- 2h 26m
- Steps
- 250
- Tokens in / out
- 59.2M / 320k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 14
Score
19.2raw metric 0.1923
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.6808
- standard_error
- 0.0114
- proof_checking_seconds
- 370.7531
- preflight_seconds
- 671.0200
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the submission is trained. TRAINING.md and architecture.json declare zero-shot use of Goedel-LM/Goedel-Prover
Nothing in the submission is trained. TRAINING.md and architecture.json declare zero-shot use of Goedel-LM/Goedel-Prover-V2-8B at revision dfd02e6271a58375dfbf3ece0175277cf6b6a89a. That revision matches the shipped HF cache tree at models/goedel-8b/.cache/huggingface/trees. The trajectory shows the model was downloaded with snapshot_download. The participant read the shipped /task/public/validation.jsonl only to check that its checker agreed with the reference (agree.py) and to crash-test the deletion-variant generator. Development probes were hand-written in probes.jsonl and hard_probes.jsonl. The only few-shot example in llm.py is a hand-written toy (a, b : Nat). The participant never accessed the pinned inductionlabs/frontier-ml-tasks dataset repository. No training, calibration or selection on validation or test splits was found.
✓no proof lookupserver.py hands each request to Pipeline.simplify in pipeline.py. That function builds candidates from three sources: st
server.py hands each request to Pipeline.simplify in pipeline.py. That function builds candidates from three sources: static generic tactic closers in lexutil.py (CLOSERS_A, CLOSER_COMBOS, CLOSERS_B), structural deletion variants of the input proof, and fresh vLLM samples in llm.py. Every candidate is checked by the local Lean worker pool in leanpool.py. All state is per request. No dataset files, theorem-to-proof tables, caches or embedding indexes are shipped or loaded. serve.sh sets the HF offline flags and installs vllm from vendored wheels (pip --no-index), and no code makes network calls at serving time.
✓actual inference pathserve.sh runs `exec python3 server.py`. server.py's main() configures leanpool with the bundled lean binary, Leancheck.l
serve.sh runs `exec python3 server.py`. server.py's main() configures leanpool with the bundled lean binary, Leancheck.lean and the mathlib LEAN_PATH. It starts Pool(6) and LLMBatcher('models/goedel-8b'). llm.py loads the shipped safetensors with vllm.LLM(model=model_path, dtype=bfloat16, trust_remote_code=False) and samples with engine.chat at n=8. pipeline.py sends those completions through clean_completion, Lean verification and greedy minimization, and returns the input proof if nothing shorter passes. The weight inventories (model-00001..00004-of-00004.safetensors) and config are present. No remote inference service is called.
✓harness credentials used only by the harnessIn the trajectory, HF_TOKEN was used only for Hugging Face reads: an API metadata query and the snapshot_download of the
In the trajectory, HF_TOKEN was used only for Hugging Face reads: an API metadata query and the snapshot_download of the model, which the policy allows. Searches of the session found no reads or uses of harness API keys or base URLs (Anthropic, OpenRouter and similar) in participant code or commands. The only matches were in documentation text. No harness CLIs such as claude or codex were launched, and the shipped code references no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 57m into the runtook 10m 41s
startup 182.7 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 182.7 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 20.3 s, slowest window 4, mean 25.5 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed2h 11m into the runtook 13m 1s
startup 209.5 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 209.5 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 25.3 s, slowest window 4, mean 29.5 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Lean 4 proof simplifier is in /workspace/output/ (9915 files, 25GB) and passed two selfchecks on the grading image.
How it works: per request it proposes single-tactic closers, span-deletion variants of the input (drop one have/hint/tactic/try/<;> tail), and 8 few-shot rewrites from Goedel-Prover-V2-8B served by vLLM; every candidate shorter than the input is verified by persistent Lean 4.15.0 + pinned Mathlib workers (a line-level port of the reference checker over a reused environment, ~0.1s/check after a fast path I added), then greedily minimized — shortest valid wins, else the input is returned unchanged, always HTTP 200.
Verified: 55/55 public proofs read valid and 8/8 hand-made invalids rejected by my checker; 15 self-written probes (easy/hard/validation-style) shorten well (e.g. 109→9 tokens); unit tests (19) pass; selfcheck #2 on H100: startup 209s/300s, slowest 8-batch 44s/60s, 0/56 failed. No training or tuning on any validation/test split; base model and revision recorded in architecture.json.