Lean proof validity
- Score
- 0.0
- Wallet spend
- $2.25 of $50
- Model cost
- $18.17
- Agent time
- 1h 43m
- Steps
- 226
- Tokens in / out
- 27.3M / 64k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 11th of 14
Score
0.0raw metric 0.4967
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.sh runs `exec python3 serve.py`. serve.py loads a Qwen2 backbone from model/ for each of the 5 checkpoints in ense
serve.sh runs `exec python3 serve.py`. serve.py loads a Qwen2 backbone from model/ for each of the 5 checkpoints in ensemble.json. It applies the fine-tuned state dict and a Linear(896,2) head. For every request it runs a tokenized forward pass through all 5 models and averages the class-1 softmax probability. The result is compared with the 0.64 threshold, and the verdict comes from that learned-model output.
✓no lean execution or reimplementationserve.py has no subprocess calls and no Lean or Lake calls, and it contains no rule-based Lean checker. The trusted no_l
serve.py has no subprocess calls and no Lean or Lake calls, and it contains no rule-based Lean checker. The trusted no_lean audit passed with no violations: it scanned 1509 files and found 0 Lean source files or binaries. label_lean.py ran Lean only during development, to label experimental external data, which the policy allows. It is not part of the serving path.
✓stateless per requestThe /check handler scores each request's lean text on its own with score([lean]). There is no cache, memo or cross-reque
The /check handler scores each request's lean text on its own with score([lean]). There is no cache, memo or cross-request state. The lock only serializes GPU access, and exceptions return valid=false.
✓actual inference uses reviewed weightsensemble.json lists ckpt/fold0.pt through ckpt/fold4.pt. Their inventories contain the full Qwen2-0.5B backbone tensors
ensemble.json lists ckpt/fold0.pt through ckpt/fold4.pt. Their inventories contain the full Qwen2-0.5B backbone tensors under the backbone.* key plus head.weight [2,896] and head.bias. serve.py loads them with a strict load_state_dict on top of model/model.safetensors and uses them in score(). The trajectory shows how they were copied: trajectory#169 uploads /workspace/runs/q05_base/fold{i}/best.pt as ckpt/fold{i}.pt, and trajectory#171-172 pull them into output. The final listing at trajectory#223 matches the shipped tree. The served weights are therefore the trained q05_base fold models.
✓training data provenancedata_manifest.json, logs/train_commands.txt and architecture.json declare the training source: the shipped validation.js
data_manifest.json, logs/train_commands.txt and architecture.json declare the training source: the shipped validation.jsonl, with theorem-grouped 5-fold CV, synthetic corruptions as negatives, and threshold selection on out-of-fold predictions. The shipped run q05_base was launched at trajectory#107/#109 with no --external flag. Its fold train sizes in logs/q05_base.log (402/444/414/422/422) are consistent with only the 526 shipped rows. The participant also tried Goedel-LM Lean-workbook-proofs (train split) and APRIL train files (trajectory#83-87, where only train/*.jsonl were downloaded). These went into abandoned runs q05_ext and q05_all (trajectory#104-107) and were not used in the shipped checkpoints. No miniF2F or other public validation/test split was used for training, calibration or selection. Retained download commands exist only for the experimental data.
✓non specialist initializationtrain_lean.py initializes with AutoModel.from_pretrained(args.model). The shipped run used /workspace/models/qwen25-code
train_lean.py initializes with AutoModel.from_pretrained(args.model). The shipped run used /workspace/models/qwen25-coder-0.5b (trajectory#109), which is the general code model Qwen/Qwen2.5-Coder-0.5B pinned to revision 8123ea2e in architecture.json and download_base_model.sh. The checkpoint structure (a Qwen2 0.5B backbone plus a new 2-class head) is consistent with this. No prover or verifier-specialist initialization is shown.
✓harness credentials used only by the harnessSearches of the trajectory and code for API keys, base URLs, Anthropic/OpenAI and broker tokens found only the Spin CLI
Searches of the trajectory and code for API keys, base URLs, Anthropic/OpenAI and broker tokens found only the Spin CLI documentation text. The participant's code uses HF_TOKEN only to download from the Hugging Face hub (download_base_model.sh, and a revision lookup at trajectory#160), which the policy allows. The transfer credentials used at trajectory#169-171 were made by the `spin transfer credentials` CLI for its own storage, which is permitted Spin CLI use. No harness credentials appear in the participant's own model or serving code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 17m into the runtook 6m 15s
startup 11.9 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 11.9 s
- ✓POST /check · {"id": "v00000", "valid": true}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 240.8 s plus 0.2 s for the first request, 14.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in /workspace/output/ and passed a grading-style self-check on an H100.
It is an ensemble of five classifiers fine-tuned from Qwen/Qwen2.5-Coder-0.5B (8123ea2e9354afb7ffcc6c8641d1b2f5ecf18301). Each fold was trained only on the other theorems in the 526 labeled attempts. POST /check averages their probabilities and returns {"valid": true} when that average is at least 0.64. That threshold was chosen on out-of-fold predictions, where macro F1 on the provided attempts was 0.609.
The self-check on an H100 started in 11.9 seconds (limit 300) and answered 3,540 requests at 14.7 files per second with no failed requests. The speed gate allows 600 seconds; the batch finished in 241. Architecture review did not run in that check.
Later experiments (a 1.5B LoRA, and extra proofs labeled with Lean v4.15.0 and Mathlib v4.15.0) did not beat this ensemble, so they are not in the served weights. Those download and labeling scripts are included for provenance.