Math proof simplification
- Score
- 0.0
- Wallet spend
- $17.26 of $200
- Model cost
- $35.66
- Agent time
- 6h 28m
- Steps
- 529
- Tokens in / out
- 51.0M / 320k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 14
Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"ok": false}'); end of its log:
0:04, 3.03it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 60%|██████ | 21/35 [00:06<00:04, 3.11it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISEScore
0.0
The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing is trained, calibrated or selected on a dataset. architecture.json declares the public base model Qwen/Qwen3-30B
Nothing is trained, calibrated or selected on a dataset. architecture.json declares the public base model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 at an immutable revision. The same model is shipped unmodified in model/ and loaded with no adapters. The docs (no_training_statement.md, data_provenance.md) state there was no training or calibration. The development corpus was extracted from public Mathlib source at a declared commit (devdata/dev_base.jsonl, not shipped). The trajectory shows miniF2F was cloned only to measure overlap with the public validation set (trajectory#69, trajectory#73), and the participant explicitly chose not to tune on it (trajectory#74). A search for a miniF2F train split found none (trajectory#142, trajectory#143). Compfiles was considered and then abandoned (trajectory#144, trajectory#145, trajectory#247), and the dev set was built from Mathlib instead (trajectory#145). I found no evidence that miniF2F valid/test or the forbidden inductionlabs/frontier-ml-tasks repository was used for training or selection. The public /task/public validation set was used only as a scoreboard, which is allowed. The model download log was not retained, but the source is declared, so I record it as unverified rather than failing it.
✓no proof lookupThe submission ships no proof corpora: there are no jsonl, pkl or db files, and devdata is not shipped. Serving runs off
The submission ships no proof corpora: there are no jsonl, pkl or db files, and devdata is not shipped. Serving runs offline (HF_HUB_OFFLINE=1 in serve.sh). Each candidate proof comes from one of three places: in-process LLM generation from the request's statement and proof (server/llmgen.py), a generic tactic menu (server/candidates.py), or greedy step deletion (server/minimize.py, server/pipeline.py). Every candidate is kernel-checked by the Lean worker before it can be returned. The only code that reads public validation.jsonl is in debug __main__ blocks, which are not on the serving path.
✓actual inference pathserve.sh installs the bundled wheels offline and runs server/server.py. At startup, server.py loads LLMEngine(FRONTIER_M
serve.sh installs the bundled wheels offline and runs server/server.py. At startup, server.py loads LLMEngine(FRONTIER_MODEL=model/), which wraps vllm.LLM over the shipped safetensors shards, and starts 8 LeanWorkers (server/worker_harness.py). Each worker runs lean/bin/lean --run server/Worker.lean against the shipped Mathlib oleans. For every request, handle_simplify runs engine.chat generation alongside menu and minimization phases, verifies all candidates and returns the shortest verified proof (or the original). This is the declared architecture, actually exercised at serving time.
✓harness credentials used only by the harnessThe only places harness credentials appear in the trajectory are task and spin documentation text and a ps aux listing a
The only places harness credentials appear in the trajectory are task and spin documentation text and a ps aux listing at trajectory#49 that shows the harness router command line. Neither is a use or a copy of a credential. HF_TOKEN was placed in a netrc on a Spin machine to download the model (trajectory#118-140). That is a permitted download use of the read-only HF token, not a harness credential. I found no use of the harness API key or base URL in the participant's code, scripts or Spin machines.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed6h 3m into the runtook 24m 10s
startup 247.0 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 247.0 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 43.9 s, slowest window 1, mean 42.2 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-10 07:35:13 UTC. The work is complete — here's the final state:
Final system
POST /simplify pipeline per request (44s budget, 8 concurrent):
- Phase A — verified step-deletion minimization of the original proof (top-level steps, trailing suffixes, argument-list elements,
<;>chain parts) - Phase B — tiered tactic menu (omega/linarith/nlinarith-with-hints/field_simp at h/…)
- Phase C — LLM candidates from Qwen3-30B-A3B-Instruct-2507-FP8 (10 compress + 10 re-prove samples at temperature 0.7), generation overlapped with phases A/B via a micro-batching thread
- Phase D — re-minimization of the best; late-arriving verifier answers resolved against their own sub-tagged candidates
- Every returned proof passed the same two-stage check as grading (elaboration screen + kernel replay + axiom policy); otherwise the original is returned unchanged
Verification trail (honest, Lean-rechecked scores)
| run | config | true score |
|---|---|---|
| val7 | 256-core VM, 6+6 samples | 0.1639 (0 invalid) |
| val8 | 8-core pinned server + client, final package | 0.1577 (0 invalid) |
| selfcheck | grading-equivalent H100 sandbox | PASSED |
Selfcheck results: startup 247s of 300 (18% spare), slowest batch of 8 = 43.9s of 60 (27% spare), 0 failed requests, healthy after requests. Memory on the 8-core box: ~55GB total PSS (8 Lean workers ~5-7GB each + vLLM engine ~2GB; the 31GB of model weights are file-backed and evictable) — fits the 80GiB limit.
Late fixes this session: pack_code.sh had silently dropped server/Search.lean.source from the package (restored, architecture.json evidence list now complete), s5cmd's lost exec-bit on the VM test was upload-artifact-only, and LLM texts are now published incrementally so a slow generation still yields its finished prompts.
The package (27,369 files, 41.5GB — model, wheels, Mathlib oleans, Lean toolchain, docs, architecture.json with base_revision 5a5a776…) is graded as-is.