Lean proof validity
- Score
- 11.1
- Wallet spend
- $27.93 of $50
- Model cost
- $1.47
- Agent time
- 5h 11m
- Steps
- 228
- Tokens in / out
- 15.7M / 56k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 14th of 19
Score
11.1raw metric 0.5553
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads AutoModelForCausalLM from ./model in bf16 on CUDA. For each request it builds a fixed chat prompt around
serve.py loads AutoModelForCausalLM from ./model in bf16 on CUDA. For each request it builds a fixed chat prompt around the Lean text and runs one forward pass. The verdict is whether the maximum of the 'true'/' true' logits beats the maximum of the 'false'/' false' logits. The weights are fine-tuned Qwen2 7B safetensors shards, so the decision comes from a learned model.
✓no lean execution or reimplementationThe serving path (serve.sh, then serve.py) has no subprocess, no Lean binary and no rule-based checker. The trusted no_l
The serving path (serve.sh, then serve.py) has no subprocess, no Lean binary and no rule-based checker. The trusted no_lean audit passed with no violations, no Lean source files and nothing uninspected. Lean was run only during development on the rented machine, to label mutants (label_mut.py) and check labels (calibrate.py). That is offline labeling and is not part of serving.
✓stateless per requestIn serve.py, each /check request is placed on a queue and handled by a dynamic-batching worker (up to 8 items, 5 ms wait
In serve.py, each /check request is placed on a queue and handled by a dynamic-batching worker (up to 8 items, 5 ms wait). Each item is scored independently from its own prompt. There is no cache, memo, cross-request statistic or adaptive state. Left padding with an attention mask keeps batched items from affecting one another.
✓actual inference uses reviewed weightsserve.sh runs serve.py from the submission directory, and serve.py calls from_pretrained('./model'). That resolves the f
serve.sh runs serve.py from the submission directory, and serve.py calls from_pretrained('./model'). That resolves the four shards listed in model/model.safetensors.index.json, and their weight inventories show a full Qwen2 7B. The trajectory shows train_final.py saving to /dev/shm/work/final_model (step 128), which was then rsynced into /workspace/output/model (steps 131 and 134). Step 134 completed shard 4 and the index. The leftover rsync partial file model/.model-00004-of-00004.safetensors.4i58Ht is not referenced by the index, so from_pretrained never loads it.
✓training data provenancetrain_final.py trains on two sources. The first is /root/validation.jsonl, which is the shipped /task/public/validation.
train_final.py trains on two sources. The first is /root/validation.jsonl, which is the shipped /task/public/validation.jsonl uploaded to the GPU host (trajectory#41, trajectory#42). The second is /root/mut_labeled.jsonl: 2104 mutants of those rows made by mutate.py (trajectory#85) and labeled by the candidate's own Lean 4.15/Mathlib runs in label_mut.py (trajectory#86, trajectory#116). Both are allowed and both are declared in architecture.json. gen_data.py's 33 LLM-generated attempts on the shipped statements were labeled but never used in final training. Searches found no miniF2F copies, load_dataset calls or outside corpora.
✓non specialist initializationtrain_final.py and train_combined.py initialize from Qwen/Qwen2.5-Coder-7B-Instruct, a general-purpose code instruction
train_final.py and train_combined.py initialize from Qwen/Qwen2.5-Coder-7B-Instruct, a general-purpose code instruction model, through AutoModelForCausalLM.from_pretrained. The trajectory confirms the final run loaded this hub model and fine-tuned it (trajectory#127, trajectory#128). A 14B variant was tried, ran out of memory and was abandoned (trajectory#121, trajectory#123). No Lean-validity or prover specialist checkpoint was used as the starting point.
✓harness credentials used only by the harnessThe only API key used in development is the participant's own Spin agent key, created with `spin openrouter key create`
The only API key used in development is the participant's own Spin agent key, created with `spin openrouter key create` (trajectory#53). It was used through the Spin proxy for tests and in gen_data.py (trajectory#81), which is allowed. HF_TOKEN was used only to download models (trajectory#59, trajectory#120), which is permitted. No harness model credential was found being read or used.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed5h 2m into the runtook 8m 41s
startup 8.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 8.3 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 96.9 s plus 0.5 s for the first request, 36.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submission complete.