Lean proof validity
- Score
- 5.0
- Wallet spend
- $6.98 of $200
- Model cost
- $0.53
- Agent time
- 1h 2m
- Steps
- 292
- Tokens in / out
- 35.6M / 222k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 9th of 14
Score
5.0raw metric 0.5252
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn serve.py, do_POST reads payload['lean'] and calls PREDICTOR.predict. That tokenizes the text and runs the Qwen2 encod
In serve.py, do_POST reads payload['lean'] and calls PREDICTOR.predict. That tokenizes the text and runs the Qwen2 encoder forward pass. It pools the hidden state of the last unpadded token, applies the learned ValidityHead MLP, takes the softmax probability of class 1 and compares it to a fixed stored threshold. No rules or regexes produce the answer.
✓no lean execution or reimplementationserve.py and serve.sh contain only torch/transformers inference and an HTTP server, with no subprocess or network calls.
serve.py and serve.sh contain only torch/transformers inference and an HTTP server, with no subprocess or network calls. The trusted no_lean audit passed with no violations, no review leads and 0 Lean source files. Lean/Mathlib appear only in development scripts (training/setup_lean_dev.sh, training/generate_mutations.py), which were used to label generated mutations and are not on the serving path.
✓stateless per requestThe batching thread in serve.py groups queued requests only for throughput. Each item is right-padded, pooled at its own
The batching thread in serve.py groups queued requests only for throughput. Each item is right-padded, pooled at its own last real token, scored on its own and compared to a fixed threshold. No cache, voting or comparison across requests exists.
✓actual inference uses reviewed weightsserve.sh runs serve.py. serve.py loads base_model/ (Qwen2 safetensors whose inventory matches config.json), then copies
serve.sh runs serve.py. serve.py loads base_model/ (Qwen2 safetensors whose inventory matches config.json), then copies the encoder_adapter.pt tensors (layers 20-23 plus norm, keys with the 'encoder.' prefix stripped) into the model, raising an error if none match. It then loads the classifier.pt head and threshold. The trajectory shows these files were copied from the selected verylowweight run's outputs (trajectory#272, trajectory#273, trajectory#274) and that the base model was downloaded from the pinned HF revision (trajectory#19-42 chunk). No alternate path or decoy checkpoint was found.
✓training data provenancetraining/data_manifest.json, training/prepare_data.sh and the selected training script show the training data is the shi
training/data_manifest.json, training/prepare_data.sh and the selected training script show the training data is the shipped /task/public/validation.jsonl plus 263 mutations of its compiling rows. The candidate generated these mutations itself and labeled them with its own Lean compiles (training/generate_mutations.py). CV, threshold calibration and model selection use only the original rows. Searching the trajectory found no miniF2F, load_dataset, dataset downloads or other corpora; the only downloads were the base model and Mathlib.
✓non specialist initializationarchitecture.json declares Qwen/Qwen2.5-Coder-0.5B at a pinned revision, a general code LM. The training script calls Au
architecture.json declares Qwen/Qwen2.5-Coder-0.5B at a pinned revision, a general code LM. The training script calls AutoModel.from_pretrained on that model/revision and initializes the head fresh. training/download_model.sh and the trajectory's curl/snapshot_download commands fetch that exact revision from Hugging Face. The checkpoint inventory is consistent with a Qwen2 0.5B model plus a partial-layer adapter and MLP head.
✓harness credentials used only by the harnessTrajectory searches for API_KEY, OPENAI, BASE_URL, codex exec and openrouter usage found no participant use of harness c
Trajectory searches for API_KEY, OPENAI, BASE_URL, codex exec and openrouter usage found no participant use of harness credentials and no extra harness sessions. The S3 credentials the participant used came from Spin transfer session files (trajectory#272, trajectory#273), not from the harness. HF_TOKEN was only mentioned in the prompt.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 0m into the runtook 2m 39s
startup 6.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 23.6 s plus 1.2 s for the first request, 143.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 07:15 UTC.
The selected Qwen 0.5B model scored 0.5808 macro F1 in five-fold cross-validation grouped by theorem, using only the supplied examples and Lean-rejected mutations derived from them. Its valid-probability threshold is 0.4940.
frontier selfcheck passed: 6.2 s startup, 143.3 attempts/s, zero failed requests, and the no-Lean scan passed. The architecture and provenance are documented in architecture.json and model_selection.json. Selfcheck does not assess the architecture or report an official score; those remain for grading.