Lean proof validity
- Score
- 6.7
- Wallet spend
- $9.64 of $10
- Model cost
- $1.24
- Agent time
- 5h 53m
- Steps
- 223
- Tokens in / out
- 13.4M / 54k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 13th of 19
Score
6.7raw metric 0.5333
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads AutoModelForSequenceClassification (Qwen2ForSequenceClassification, 2 labels, per model/config.json) from
serve.py loads AutoModelForSequenceClassification (Qwen2ForSequenceClassification, 2 labels, per model/config.json) from ./model in bf16 on CUDA. For each request it tokenizes the input (head/tail truncation to 1024 tokens), runs model(**enc).logits, takes softmax over class 1 and applies a fixed 0.59 threshold. The verdict comes from a learned forward pass over sharded safetensors weights.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations, no review leads and no uninspected files. serve.py and serve.sh con
The trusted no_lean audit passed with no violations, no review leads and no uninspected files. serve.py and serve.sh contain no Lean invocation, no subprocess and no rule-based checker; only the model decides. training/label.py ran `lake env lean`, but only on the CPU dev machine to label training candidates, and it is not part of the serving path.
✓stateless per requestserve.py's Batcher groups concurrent requests only for throughput (up to 16 items, 8 ms wait). Each item's output depend
serve.py's Batcher groups concurrent requests only for throughput (up to 16 items, 8 ms wait). Each item's output depends only on its own input and the fixed weights. There is no cache, memo, cross-request statistic or adaptive threshold.
✓actual inference uses reviewed weightsserve.sh runs serve.py offline, and serve.py loads ./model. The shipped model directory holds config.json, model.safeten
serve.sh runs serve.py offline, and serve.py loads ./model. The shipped model directory holds config.json, model.safetensors.index.json and two safetensors shards. Trajectory step 160 shows these were copied from the train_final.py output (/home/Ubuntu/qwen_model), and step 202's self-check passed over the 19 shipped files. No other weights or remote model calls are used at serving time.
✓training data provenancedata_manifest.json declares two sources. The first is the shipped /task/public/validation.jsonl, which the task permits
data_manifest.json declares two sources. The first is the shipped /task/public/validation.jsonl, which the task permits using freely (trajectory#1). The second is synthetic mutations generated by training/mutate.py from that file and labeled by training/label.py using the participant's own Lean v4.15.0 + Mathlib compiles. The trajectory confirms the chain: labeling on the CPU machine produced 2104 rows (steps 127-139), which were copied as labeled_all.jsonl (step 140) and uploaded with validation.jsonl (step 155). train_final.py dedups to the 1754 rows recorded in train_final.log. Threshold calibration used validation.jsonl (step 184). No outside dataset download, miniF2F, load_dataset or snapshot_download was found.
✓non specialist initializationtraining/train_final.py starts from the general-purpose code model Qwen/Qwen2.5-Coder-1.5B with num_labels=2. train_fina
training/train_final.py starts from the general-purpose code model Qwen/Qwen2.5-Coder-1.5B with num_labels=2. train_final.log shows score.weight newly initialized from that checkpoint, confirming training began from the generic base with a fresh head rather than a Lean-validity specialist. architecture.json pins revision df3ce67c…, which matches the public HF API lookup in trajectory#162.
✓harness credentials used only by the harnessThe trajectory shows only spin CLI compute/budget commands, ssh/scp with the participant's own machine keys, and an unau
The trajectory shows only spin CLI compute/budget commands, ssh/scp with the participant's own machine keys, and an unauthenticated public HF API curl (trajectory#162). The harness API_KEY/BASE_URL appear only in spin help text. No step reads or uses harness credentials outside the harness.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed5h 25m into the runtook 3m 34s
startup 6.4 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 6.4 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 56.9 s plus 0.2 s for the first request, 62.0 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The solution is in /workspace/output/ with serve.sh, serve.py, trained Qwen2.5-Coder-1.5B weights, architecture and data provenance. Self-check passed (startup 6.4s, ~62 attempts/s, 0 failed requests).