Lean proof validity
- Score
- 36.3
- Wallet spend
- $9.89 of $10
- Model cost
- $7.00
- Agent time
- 3h 1m
- Steps
- 210
- Tokens in / out
- 21.3M / 307k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 4th of 14
Score
36.3raw metric 0.6817
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passEach prediction comes from learned models. server.py's Predictor runs the Qwen3 ValidityModel (modeling.py: AutoModel in
Each prediction comes from learned models. server.py's Predictor runs the Qwen3 ValidityModel (modeling.py: AutoModel in bf16, last-token and masked-mean pooling, Linear score head loaded from model/head.safetensors). It also runs the frozen Llama secondary encoder with FrozenHead (frozen_modeling.py: LayerNorm and MLP, loaded from secondary/head.safetensors). The two probabilities are mixed with weight 0.4 and thresholded at 0.5 (model/classifier_config.json). No rules, retrieval or cache sit on the decision path.
✓no lean execution or reimplementationThe trusted no_lean audit passed: 103 files scanned, no violations, nothing uninspected. server.py has no subprocess or
The trusted no_lean audit passed: 103 files scanned, no violations, nothing uninspected. server.py has no subprocess or Lean invocation. The only Lean REPL use is the development-time labeling Compiler in training/build_data.py, which the policy allows. serve.sh only sets offline environment variables and runs server.py.
✓stateless per requestThe /check handler sends each request to a dynamic batching queue (up to 64 requests within an 18 ms window) purely for
The /check handler sends each request to a dynamic batching queue (up to 64 requests within an 18 ms window) purely for throughput. Each item is scored independently from its own text (preprocess.py normalize/prompt). There is no answer cache, no comparison between requests and no persistent state that affects predictions.
✓actual inference uses reviewed weightsserve.sh executes server.py. server.py loads model/ (Qwen3 shards, per the inventory) plus model/head.safetensors, and s
serve.sh executes server.py. server.py loads model/ (Qwen3 shards, per the inventory) plus model/head.safetensors, and secondary/ (Llama shards) plus secondary/head.safetensors. These are the reviewed tensor inventories. training_artifacts/ and reasoning.py are not imported or loaded by the server. At trajectory#203/#204 the participant extracted the final deployment tar into the output directory and checked the model/ and secondary/ shard indices.
✓training data provenancedata_manifest.json declares the training sources: the 526 provided attempts and Goedel-LM/Lean-workbook-proofs (train sp
data_manifest.json declares the training sources: the 526 provided attempts and Goedel-LM/Lean-workbook-proofs (train split at a pinned revision). Augmented rows were labeled by a development-time Lean v4.15.0 REPL (training/build_data.py), and training/download.py retains the snapshot_download call. miniF2F (yangky11/miniF2F-lean4 Test/Valid at a pinned revision) is used only to build exclusion keys in forbidden_keys(). The participant noticed an exclusion gap and fixed it (trajectory#95), then audited it: 488 benchmark keys, with 0 contaminated rows kept (trajectory#93, trajectory#97). training/filter_data.py re-filters banned workbook parents. The FrozenHead (training/frozen.py) is trained on the same declared data.
✓non specialist initializationarchitecture.json declares Goedel-LM/Goedel-Prover-V2-8B and deepseek-ai/DeepSeek-Prover-V2-7B at pinned revisions. Thes
architecture.json declares Goedel-LM/Goedel-Prover-V2-8B and deepseek-ai/DeepSeek-Prover-V2-7B at pinned revisions. These are allowed prover bases, not validity, critic or reward models. training/download.py and the download_secondary script in the trajectory (trajectory#83-112 range) fetch exactly these repos. model/config.json (Qwen3, 36 layers) and secondary/config.json (Llama, 30 layers, vocab 102400) match those bases. The task heads are newly trained.
✓harness credentials used only by the harnessAt trajectory#9 the participant listed environment variable names only (env | cut -d= -f1), not values. The 'codex exec'
At trajectory#9 the participant listed environment variable names only (env | cut -d= -f1), not values. The 'codex exec' text in steps 1-22 is the harness's own process seen in a ps listing. The 'Authorization' hit in steps 43-82 (trajectory#67) is the participant reading the source of the validation tool's cli.py, not making an API call. No participant use of OPENAI or other harness keys was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 51m into the runtook 8m 6s
startup 13.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 13.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 123.2 s plus 0.1 s for the first request, 28.7 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the offline model, serve.sh, weights, and training evidence.
Production selfcheck passes:
- Startup: 13.7 seconds
- Throughput: 28.7 files/second on one H100
- Failed requests: 0 of 3,540
- No-Lean scan: passed