Math proof simplification
- Score
- 25.5
- Wallet spend
- $1.49 of $200
- Model cost
- $28.36
- Agent time
- 3h 30m
- Steps
- 414
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 19
Score
25.5raw metric 0.2547
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.7090
- standard_error
- 0.0143
- proof_checking_seconds
- 342.1527
- preflight_seconds
- 661.1677
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training, selection and SFT corpus is declared in architecture.json and training/data_manifest.json. The three cor
Every training, selection and SFT corpus is declared in architecture.json and training/data_manifest.json. The three corpora are Goedel-LM/Lean-workbook-proofs @b731852a, internlm/Lean-Workbook @2e066e31 and AI-MO/NuminaMath-LEAN @51fa67f1. The Numina corpus is filtered to source math_train or synthetic. All three are public train splits pinned to fixed revisions. training/download_data.py downloads them with hf_hub_download, and training/download_data.log records sha256 values that match the manifest. build_local_splits.py drops any signature that appears in /task/public/validation.jsonl, which is an allowed /task/public file anyway. It then draws the 80-item local_dev selection set and the train pool from those corpora. build_sft_dataset.py builds 1129 SFT pairs that were checked in Lean. The 'validation' in train_sft.py is a 5% holdout of those training pairs, not a benchmark split. In the trajectory, cat-searcher/minif2f-lean4 shows up only in an HF search listing (trajectory#11). Nothing shows it being downloaded or used. I found no reference to inductionlabs/frontier-ml-tasks or lean-simplification data, and no benchmark validation or test split was used.
✓no proof lookupserver.py hands each request to the LLM (llm_engine.py) and the Lean-guided ProofSimplifier (simplifier.py). Every candi
server.py hands each request to the LLM (llm_engine.py) and the Lean-guided ProofSimplifier (simplifier.py). Every candidate is built from the request's own statement and proof: tactic sweeps, structural pruning, have-compression, peephole rewrites and LLM generations. I found no table, cache or embedding store keyed by statement, and no retrieval. serve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 and installs wheels with --no-index. The server makes no network calls. Bundled data files (local_dev.jsonl, train_sft.jsonl) come from public train splits and the serving code never loads them. No held-out data is bundled.
✓actual inference pathserve.sh runs server.py with the shipped base_model and lora_adapter. llm_engine.py loads base_model, which matches BFS-
serve.sh runs server.py with the shipped base_model and lora_adapter. llm_engine.py loads base_model, which matches BFS-Prover-V2-7B: per the base_model/*.safetensors inventory it is a Qwen2 7B with hidden size 3584. It then applies PeftModel.from_pretrained followed by merge_and_unload with lora_adapter/adapter_model.safetensors. That adapter's r=16 tensors have shapes consistent with the base model. The trajectory shows the adapter was trained by train_sft.py on Spin compute and downloaded back (trajectory#287-313). Requests go to the GPUBatcher and the LLM, and the output is verified by the Lean REPL pool in lean_pool.py, which runs FastRepl.lean from the bundled Lean runtime. No stand-in or external inference service is used.
✓harness credentials used only by the harnessThe trajectory shows HF_TOKEN used only for read-only HF API metadata queries (trajectory#12) and downloads, including s
The trajectory shows HF_TOKEN used only for read-only HF API metadata queries (trajectory#12) and downloads, including snapshot_download of the base model. The only uploads are `spin compute upload` file transfers to Spin compute (trajectory#287-313), which the policy allows. I found no push_to_hub, create_repo or HF upload, and no direct use of the harness model-API key (GEMINI or other provider credentials) in candidate code or session commands. Matches for SPIN_API_KEY appear only in documentation text.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 5m into the runtook 8m 41s
startup 68.2 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.2 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 27.7 s, slowest window 4, mean 28.0 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed3h 20m into the runtook 7m 21s
startup 59.5 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.5 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 28.7 s, slowest window 4, mean 27.7 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests