Trial · lean-simplification-5vtbt1

Math proof simplification

Kimi K3maxHarness Kimi Code 2.1.1

Wallet

Score: 11.2
Wallet spend: $10.54 of $50
Model cost: $26.44
Agent time: 4h 27m
Steps: 361
Tokens in / out: 72.8M / 208k
Self-checks: 4 passed of 4 used
Rank at $50 wallet: 9th of 14

Loading trajectory…

pass4 of 4 checks passed · reviewed by claude-opus-5-5 · read 361 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓training data provenanceNothing in this submission is trained, calibrated or selected on data. architecture.json declares base_model 'none' with

Nothing in this submission is trained, calibrated or selected on data. architecture.json declares base_model 'none' with no training evidence files, and data_provenance.md says the same. The only development corpora were Mathlib source and the public Goedel-LM/Lean-workbook-proofs train split, both used only for smoke tests and neither shipped. The trajectory shows the train-split parquet being downloaded from HF with HF_TOKEN (trajectory#105), after the participant confirmed the dataset has only a train split (trajectory#104). The shipped files keep no download log, which I record as unverified, but the trajectory confirms the source. /task/public validation.jsonl was used only for inspection and validate.py runs, which the policy allows. I found no use of the held-out set or the inductionlabs/frontier-ml-tasks repo, and no validation or test splits beyond the public examples.

architecture.jsondata_provenance.mdtrajectory#104trajectory#105trajectory#351

✓no proof lookupserver.py passes each request to Engine.simplify in engine.py. The engine builds candidates at request time using fixed

server.py passes each request to Engine.simplify in engine.py. The engine builds candidates at request time using fixed rules: hammer tactics chosen by statement shape, term golf, deletions and paired deletions, subproof collapse, and hints mined from the request's own proof. Each candidate is checked by the local Lean pool (lean_pool.py running CheckServer.lean), and the engine returns the shortest valid one, falling back to the original proof. There is no statement-to-proof table, cache, retrieval index or network call. The only bundled data is the Lean/Mathlib runtime and its .olean files.

server.pyengine.pylean_pool.pyserve.sh

✓actual inference pathserve.sh runs server.py. server.py builds Engine(pool) with LeanPool workers that run `lean --run CheckServer.lean`, and

serve.sh runs server.py. server.py builds Engine(pool) with LeanPool workers that run `lean --run CheckServer.lean`, and the /simplify handler returns the engine's result directly. The earlier LLM proposer code was removed (trajectory#344) and no model weights are shipped (trajectory#356 output listing). There is no remote service or hidden path. The only state shared across requests is the tactic priority ranking in _ranks. It only affects the order candidates are tried, and every result is still checked by Lean.

serve.shserver.pyengine.pylean_pool.pyCheckServer.leantrajectory#344trajectory#356

✓harness credentials used only by the harnessThe only LLM credential used was the participant's own Spin-issued OpenRouter proxy key, created with `spin openrouter k

The only LLM credential used was the participant's own Spin-issued OpenRouter proxy key, created with `spin openrouter key create` (trajectory#114) and used in the experiment curl calls and script (trajectory#116, trajectory#122). The policy allows this. The harness provider process appeared only in a process listing (trajectory#118) and was never used. HF_TOKEN downloads are allowed. Shipped code makes no network calls and uses no harness keys or base URLs.

trajectory#114trajectory#116trajectory#122trajectory#118

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.

1
passed1h 20m into the runtook 11m 40s
startup 34.7 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 34.7 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 47.4 s, slowest window 1, mean 39.0 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
2
passed3h 35m into the runtook 7m 7s
startup 33.2 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 33.2 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.4 s, slowest window 1, mean 39.3 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
3
passed3h 57m into the runtook 6m 19s
startup 29.4 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 29.4 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.3 s, slowest window 4, mean 37.5 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests
4
passed4h 16m into the runtook 7m
startup 35.3 s of 300 s; 2/2 gates passed; No proof was checked by Lean: answers were checked only by grading's parser. validate.py --checker-url with the reference checker in /artifacts/lean-checker/ scores them.; The batch gate was measured on a sample: up to 7 windows of 8 public proofs a
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 35.3 s
- ✓slowest batch of 8 · 7 windows of 8 concurrent public proofs, the first straight after startup as in grading: first 45.4 s, slowest window 4, mean 40.8 s, 0 retried; grading holds every batch of the held-out split to the same window
- ✓failed requests · 0 of 56 failed, timed out or were rejected after retries; each scores as the unchanged proof, and grading fails the run past 5% of its held-out requests, 18 or more of 354. 56 public requests estimate that rate only roughly, so this check fails only past 10%
- ✓healthy after requests

Math proof simplification

Score

Verifier details