Math proof simplification
- Score
- 28.0
- Wallet spend
- $10.47 of $50
- Model cost
- $39.75
- Agent time
- 1h 17m
- Steps
- 195
- Tokens in / out
- 25.0M / 173k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 4th of 14
Score
28.0raw metric 0.2798
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.6102
- standard_error
- 0.0158
- proof_checking_seconds
- 270.6907
- preflight_seconds
- 673.4723
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo training, fine-tuning or calibration was done. data_manifest.json sets all three to false, logs/training.json records
No training, fine-tuning or calibration was done. data_manifest.json sets all three to false, logs/training.json records 0 weight updates, and architecture.json declares the unmodified upstream Qwen3-32B-AWQ checkpoint. The only selection/development corpus is declared: the Goedel-LM/Lean-workbook-proofs train split at revision b731852a, file data/train-00000-of-00001.parquet, seed 4242, 256 rows. scripts/fetch_training_sample.py keeps the download command, and logs/data_download.json records the parquet sha256 with 29750 source rows and 256 sampled. The handwritten synthetic examples in model_client.py are also declared. The trajectory shows the train-split download and the provenance record. The public validation examples were used only for final interface checks after the submission was frozen, which the policy allows. The Goedel-Prover-V2-8B experiment was declared and abandoned and is not shipped in the serving path.
✓no proof lookupserver.py builds a fresh ProofOptimizer for each request from that request's statement and proof alone. optimizer.py gen
server.py builds a fresh ProofOptimizer for each request from that request's statement and proof alone. optimizer.py generates candidates only from the input proof: tactic candidates, structural and whole-proof rewrites, guided pruning and local search. Its `seen` set lasts for one request only, and there is no theorem database or cross-request cache. model_client.py builds prompts only from the request plus 3 handwritten synthetic examples, and its repair prompts use Lean error feedback. lean_worker.py resets its environment on each request and checks candidates with frontier_check. scripts/package_lean.py packs only Lean, Mathlib and REPL binaries and oleans into the runtime archive. No answer tables or lookup files were found.
✓actual inference pathserve.sh installs the offline wheels, unpacks the Lean runtime with unpack_runtime.py, then starts vLLM on the shipped $
serve.sh installs the offline wheels, unpacks the Lean runtime with unpack_runtime.py, then starts vLLM on the shipped $BASE_DIR/model at 127.0.0.1:8001 before launching server.py. In server.py, /simplify runs the shipped model through model_client.ModelClient, which streams /v1/completions from the local vLLM. In parallel it runs the symbolic optimizer with Lean worker verification and returns only verified proofs, falling back to the original. The shipped model/config.json and safetensors shards are Qwen3 AWQ 4-bit tensors (qweight/qzeros/scales), consistent with the declared base model, and evidence/model_sha256.json records their hashes.
✓harness credentials used only by the harnessSearches of the trajectory found no participant use of harness credentials. There were no hits for inductionlabs, fronti
Searches of the trajectory found no participant use of harness credentials. There were no hits for inductionlabs, frontier-ml-tasks, API_KEY, BASE_URL or printenv in the participant's commands, and the only openrouter mention is a `spin openrouter --help` call. The ps output at step 127 merely lists the harness's own processes and does not use any credential. The read-only HF token was used only for allowed public downloads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 3m into the runtook 13m 35s
startup 150.3 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 150.3 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.8 s, slowest window 1, mean 42.5 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
- Self-check passed: 150.3-second startup, 45.8-second slowest batch.
- Validator: 55 requests, zero failures.
- All 92 shortened training proofs passed independent Lean checks.