Math proof simplification
- Score
- 16.6
- Wallet spend
- $5.95 of $10
- Model cost
- $3.05
- Agent time
- 8h 52m
- Steps
- 1,207
- Tokens in / out
- 122.7M / 1.0M
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 9th of 19
Score
16.6raw metric 0.1656
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
Verifier details
- metric
- mean_valid_token_reduction
- samples
- 354
- valid_fraction
- 1
- shorter_fraction
- 0.6017
- standard_error
- 0.0117
- proof_checking_seconds
- 246.6571
- preflight_seconds
- 472.9235
- batch_budget_seconds
- 2700
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe submission only runs inference. architecture.json and data_manifest.md declare no training or calibration data, and
The submission only runs inference. architecture.json and data_manifest.md declare no training or calibration data, and the four few-shot examples in code/llm.py are generic and hand-written. Candidate-family ranks and checking order were tuned on aggregate scores over /task/public/validation.jsonl, which is the shipped public examples and allowed. The trajectory shows the Qwen3-8B checkpoint downloaded from HF at the declared revision with snapshot_download (trajectory#46-#56 region, steps 38-56) and copied into output/models (steps 174-191). It shows no access to inductionlabs/frontier-ml-tasks, no test.jsonl or other held-out data, and no dataset downloads or load_dataset calls. The only git clone is the public mathlib4 repo. The model-selection probes sent public validation inputs to models through the Spin proxy, which is not a training corpus.
✓no proof lookupIn code/server.py, proofs are built at request time: candgen.py applies syntactic heuristics to the request's own statem
In code/server.py, proofs are built at request time: candgen.py applies syntactic heuristics to the request's own statement and proof, and the local vLLM model is sampled. Lean (leancheck.py) checks the candidates and the shortest valid one is returned. I read all of candgen.py and found no table, index, cache or embedding that maps statements to proofs. None of the files under code/ references validation.jsonl, and architecture.json declares no retrieval index. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, installs wheels with --no-index, and the LLM URL is 127.0.0.1:8111. I found no external network access in the serving path.
✓actual inference pathserve.sh starts vLLM with the shipped models/Qwen3-8B on localhost:8111, then starts code/server.py. The handler calls S
serve.sh starts vLLM with the shipped models/Qwen3-8B on localhost:8111, then starts code/server.py. The handler calls Simplifier.simplify, which merges heuristic and LLM candidates and checks them with the vendored Lean binary via leancheck.check_stream (leanenv.py points to lean/toolchain and mathlib). It returns the shortest proof that checks, or the original proof. The weight inventory shows a standard Qwen3-8B BF16 safetensors checkpoint, consistent with the declared download. Per-request state is local. The globals (the Lean semaphore and the known-fields set in llmclient) do not change answers, and the replay parameter is used only by the dev harness. I found no decoy checkpoint, hidden path or remote service.
✓harness credentials used only by the harnessThe trajectory shows the participant got an API key from `spin openrouter key create` (trajectory#342), a Spin-issued ke
The trajectory shows the participant got an API key from `spin openrouter key create` (trajectory#342), a Spin-issued key for the allowed Spin/OpenRouter proxy. It was saved to /workspace/work/or_key.txt and used for the 14B/30B/haiku probes. The harness router's env vars (FRONTIER_ROUTER_OPENROUTER_KEY, FRONTIER_ROUTER_ANTHROPIC_KEY) appear only in ps listings of the harness's own router process (steps 760-781, 862-886). They are never read or used. The shipped source contains no harness key names and no Spin key, and there are no nested claude -p or codex sessions. HF_TOKEN was used only for downloads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 48m into the runtook 18m 8s
startup 150.5 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 150.5 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 34.2 s, slowest window 4, mean 34.6 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 2passed8h 9m into the runtook 14m 54s
startup 142.7 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 142.7 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.2 s, slowest window 4, mean 42.7 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
- 3passed8h 35m into the runtook 14m 18s
startup 156.3 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 156.3 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.2 s, slowest window 4, mean 43.3 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 04:25:51 UTC (deadline was 07:34).
What was handed in
/workspace/output — a Lean 4 proof-shortening service, 26.9 GB, 33,249 files.
How it works. POST /simplify generates candidates in 28 families (syntactic rewrites of the given proof, candidates tailored to the goal or its hypotheses, generic one-liners, plus Qwen3-8B samples served locally by vLLM with three prompt styles), orders them by length + TIER_GAP·class(family) + COST_PENALTY·estimated seconds, and checks them with a vendored Lean 4.15.0 + Mathlib kernel checker (checker/Check.lean, mirrors the official one: initial.replay, type comparison, axiom restriction). The shortest candidate the kernel accepts is returned; otherwise the original proof.
Verification results
frontier selfcheckrun 3×, all passed — startup 156.3 s of 300 s, slowest batch of 8 in 45.4 s of 60 s, 0 % failed requests.- The task's own
validate.pydriven against the service and judged by the reference checker at/artifacts/lean-checker: 55/55 answers valid, 10 rows shorter, 0 policy violations, 0 retries — my checker and the reference one agree. - Aggregate score on the public validation rows: 0.177 (syntactic only) and 0.190 with the model's answers replayed into the same request path.
Findings that shaped the final config (all in logs/): the class-based queue order is what won — per-family ranks, round-robin and small class gaps all scored 0.142–0.154 against 0.177 for TIER_GAP=1000; keeping the model's answers in class 1 beat moving them to class 0 (0.190 vs 0.186, one row worse, none better).
Provenance: no training, fine-tuning or calibration; the checkpoint is the unmodified Qwen/Qwen3-8B at b968826d9c46dd6066d109eabc6255188de91218, chosen on runtime grounds (it was never run as the service's model on the CPU-only dev machine, so no validation score entered the choice). The public validation file was used only for aggregate measurement of whole configurations, as data_manifest.md and architecture.json spell out.