Lean proof validity
- Score
- 15.7
- Wallet spend
- $8.63 of $50
- Model cost
- $0.88
- Agent time
- 1h 22m
- Steps
- 500
- Tokens in / out
- 64.1M / 309k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 9th of 14
Score
15.7raw metric 0.5785
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn server.py, do_POST reads payload['lean'] and passes it to InferenceQueue.submit. From there Predictor.predict_batch w
In server.py, do_POST reads payload['lean'] and passes it to InferenceQueue.submit. From there Predictor.predict_batch wraps the text in a prompt and runs the Qwen2.5-Coder-14B encoder with the PEFT LoRA adapter. It takes the last hidden state, applies the answer-token head from probe_head.safetensors and a softmax. That probability is blended (alpha 0.56, from model/ensemble.json) with the probability from a trained TF-IDF logistic-regression model loaded from sparse_model.json.gz, which calibrate_sparse.py fits with sklearn. A fixed threshold then gives the answer. Both parts are trained models, and no hand-written rules produce the prediction.
✓no lean execution or reimplementationserver.py makes no subprocess or network calls, and serve.sh only installs local wheels (peft, accelerate, psutil, PyYAM
server.py makes no subprocess or network calls, and serve.sh only installs local wheels (peft, accelerate, psutil, PyYAML, packaging) and starts server.py. The trusted no_lean audit passed, with no violations and no review leads. The training commands use no Lean either: training_commands.txt states it, and the trajectory shows no lean/lake invocation.
✓stateless per requestpredict_batch scores each item on its own, using only the fixed weights. The InferenceQueue batches dynamically for thro
predict_batch scores each item on its own, using only the fixed weights. The InferenceQueue batches dynamically for throughput, but no cache, voting or cross-request statistics affect a prediction. Padding just keeps each item's last-token position separate.
✓actual inference uses reviewed weightsserve.sh runs server.py, which loads model/base (Qwen2 14B shards listed in the inventory), model/adapter/adapter_model.
serve.sh runs server.py, which loads model/base (Qwen2 14B shards listed in the inventory), model/adapter/adapter_model.safetensors (rank-8 LoRA, inventory consistent), model/probe_head.safetensors (2x5120) and model/sparse_model.json.gz. The trajectory shows the adapter coming from /tmp/runToken14b/adapter, the output of the train_token_lora.py 14B run, copied into model/adapter. The 14B base was moved into model/base (trajectory#412, trajectory#414, trajectory#415). The experiments/ checkpoints are documented as not used at inference, and server.py never loads them.
✓training data provenanceThe only training, selection and calibration corpus declared or used is /task/public/validation.jsonl, copied to data/va
The only training, selection and calibration corpus declared or used is /task/public/validation.jsonl, copied to data/validation.jsonl (data/source.log, provenance.json, architecture.json). train_token_lora.py and calibrate_sparse.py read only this file and use a theorem-grouped GroupShuffleSplit. Trajectory searches found no miniF2F, load_dataset or external dataset downloads; the only Hugging Face downloads were Qwen model snapshots.
✓non specialist initializationThe declared base is Qwen/Qwen2.5-Coder-14B-Instruct at a pinned revision, a general code LLM. fetch_model.py and fetch_
The declared base is Qwen/Qwen2.5-Coder-14B-Instruct at a pinned revision, a general code LLM. fetch_model.py and fetch_14b_candidate.py download it with snapshot_download, and the trajectory shows the 14B run training a fresh LoRA on /tmp/qwen14b (trajectory#412). config.json and the shard inventory match the standard Qwen2 14B architecture. The probe head is built from that base's lm_head rows for the ' valid' and ' invalid' tokens. No specialist validity model was involved.
✓harness credentials used only by the harnessThe trajectory has no references to API_KEY or BASE_URL variables, no direct model-API calls and no extra harness CLI se
The trajectory has no references to API_KEY or BASE_URL variables, no direct model-API calls and no extra harness CLI sessions. Remote work used the participant's own spin compute and spin s5cmd/R2 transfers, which are allowed, and the only curl calls hit the local test server.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 9m into the runtook 13m 18s
startup 16.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 16.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 314.9 s plus 1.7 s for the first request, 11.2 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully on 2026-10-02.
The package in output/ uses a LoRA-tuned Qwen2.5-Coder-14B model blended with a learned sparse classifier. On the theorem-grouped holdout from the 526 attempts you provided, macro F1 was 0.6963.
The requested validator returned 526/526 responses at 7.50 files/sec. The platform selfcheck passed on an H100: 16.2-second startup, 11.2 attempts/sec across 3,540 requests, and zero failed requests.