Math proof simplification
- Score
- 7.3
- Wallet spend
- $13.96 of $50
- Model cost
- $4.73
- Agent time
- 1h 21m
- Steps
- 120
- Tokens in / out
- 13.0M / 123k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 13th of 19
Score
7.3raw metric 0.07346
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.2599
- standard_error
- 0.0088
- proof_checking_seconds
- 92.8306
- preflight_seconds
- 473.6923
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe base model is declared in architecture.json as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. The corpus is dec
The base model is declared in architecture.json as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. The corpus is declared in evidence/data_manifest.json: wp_train.jsonl (7708 rows) from leanpolish-anon/lean-proof-compression at a pinned revision. These are Goedel-Workbook SFT train-split pairs. training/download_data.py downloads the pinned archive and checks its sha256. training/upstream_build_pairs.py skips validation files and anything not in the train split, and evidence/upstream_data_filters.json records that 0 miniF2F or Putnam files were used. training/build_training.py builds the 7580/128 local train/holdout split. The trajectory shows the downloaded archive also contained miniF2F and Putnam eval item files, but the build manifest uses only wp_train.jsonl as its source. Checkpoint selection used the internal holdout (evidence/selection.json). The public validation file was used only for post-hoc reporting.
✓no proof lookupIn server.py, POST /simplify runs generic tactic candidates and vLLM samples from model_runner.py, then screens them wit
In server.py, POST /simplify runs generic tactic candidates and vLLM samples from model_runner.py, then screens them with the Lean checker in lean_runtime.py. If no candidate verifies, it falls back to the original proof. There is no theorem-keyed table, no bundled answer data (.jsonl, SQLite or pickle), and no network access at serve time: offline flags are set in serve.sh and model_runner.py.
✓actual inference pathserve.sh installs local wheels and runs exec python server.py. server.py builds ProofGenerator(ROOT/'model'), and model_
serve.sh installs local wheels and runs exec python server.py. server.py builds ProofGenerator(ROOT/'model'), and model_runner.py loads the model with vLLM LLM(model=model_path) using local files only. The shipped safetensors shards (about 8.19B parameters, lm_head [151936, 4096]) match model/model.safetensors.index.json and evidence/merged_model_manifest.json. There is no decoy checkpoint, no remote inference, and no cross-request answer state.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, API_KEY, BASE_URL, 'claude -p', codex-style launches and frontier-ml-tasks found no u
Trajectory searches for ANTHROPIC, API_KEY, BASE_URL, 'claude -p', codex-style launches and frontier-ml-tasks found no use of harness credentials by the participant. HF_TOKEN was used for Hugging Face downloads (trajectory#7), which is allowed. OpenRouter access appears only through the broker-run spin CLI (trajectory#6), which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 8m into the runtook 12m 10s
startup 176.7 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 176.7 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 36.2 s, slowest window 4, mean 39.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-24 02:48:11 UTC.
Final self-check passed:
- Startup: 176.7 s / 300 s
- Slowest batch: 51.86 s / 60 s
- Failed requests: 0 / 56
/workspace/output was submitted unchanged after the checked snapshot.