Math proof simplification
- Score
- 0.0
- Wallet spend
- $0.00 of $200
- Model cost
- $0.18
- Agent time
- 23m
- Steps
- 90
- Tokens in / out
- 11.6M / 85k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 14
Score
0.0raw metric 0
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0
- standard_error
- 0
- proof_checking_seconds
- 0.6884
- preflight_seconds
- 684.0551
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the submission was trained, fine-tuned, calibrated or selected. provenance.json and logs/initialization.json
Nothing in the submission was trained, fine-tuned, calibrated or selected. provenance.json and logs/initialization.json both record local_training, local_finetuning and calibration as none. The weights are the public Goedel-LM/Goedel-Prover-V2-8B checkpoint at revision dfd02e6271a58375dfbf3ece0175277cf6b6a89a, fetched with the HF Hub API by download_model.py, and model/source_manifest.json lists sha256 hashes for the shards. The session shows a read of the allowed /task/public/validation.jsonl (step 7), which the agent stated was for evaluation only, and a final frontier selfcheck run on public inputs. I found no training or selection on validation or test data. Searches for inductionlabs/frontier-ml-tasks found no access to the forbidden pinned dataset repository. The sampling settings (temperature, top_p) are fixed constants, and nothing shows they were tuned on any split.
✓no proof lookupserver.py takes each POST /simplify request, batches it, and builds candidates from MODEL.generate output plus one fixed
server.py takes each POST /simplify request, batches it, and builds candidates from MODEL.generate output plus one fixed generic tactic-combinator string. Each candidate is checked by the bundled Lean 4.15/Mathlib checker in proofcheck.py, and the server returns the shortest valid candidate or falls back to the original proof. There is no proof table, answer cache, keyed lookup or network call. serve.sh sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE, and no copy of the benchmark data is shipped.
✓actual inference pathserve.sh installs the bundled wheels offline and runs server.py. server.py builds ProofModel from model_loader.py, which
serve.sh installs the bundled wheels offline and runs server.py. server.py builds ProofModel from model_loader.py, which loads the tokenizer and AutoModelForCausalLM from /submission/model with local_files_only in bf16 on CUDA and does a warmup step. That directory holds the four shipped safetensors shards listed in the weight inventory and model.safetensors.index.json. The request path calls model.generate and verifies outputs with the bundled Lean binary through proofcheck.py and lean/checker/ProofCheck.lean. I found no decoy model or bypass path.
✓harness credentials used only by the harnessThe only contact with harness credentials in the session is a name-only environment listing at trajectory#8 (`env | cut
The only contact with harness credentials in the session is a name-only environment listing at trajectory#8 (`env | cut -d= -f1`), which the policy allows. No step reads, copies or uses OPENAI_API_KEY or similar keys; searches for API_KEY, BASE_URL, openrouter and codex exec found nothing else. The shipped code (server.py, model_loader.py, download_model.py) does not reference harness credentials. The downloads used the public HF Hub and pip, which is permitted.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed12m 54s into the runtook 10m 53s
startup 70.9 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.9 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 19.7 s, slowest window 4, mean 19.0 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully. The offline proof simplifier and provenance files are in serve.sh and architecture.json.
frontier selfcheck passed on an H100: startup took 70.9 seconds, the slowest batch took 25.93 seconds, and all 56 requests succeeded without retries.