Lean proof validity
- Score
- 32.5
- Wallet spend
- $28.55 of $200
- Model cost
- $29.66
- Agent time
- 3h 28m
- Steps
- 464
- Tokens in / out
- — / —
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 7th of 19
Score
32.5raw metric 0.6627
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserver.py's /check handler tokenizes the input. It then runs a forward pass through three causal LMs, each with a LoRA m
server.py's /check handler tokenizes the input. It then runs a forward pass through three causal LMs, each with a LoRA merged in (Goedel-Prover-V2-8B, DeepSeek-Prover-V2-7B and Goedel-Prover-SFT). From that pass it collects token log-prob statistics and hidden states via hooks on layers -1/-4/-8/-12. A trained ridge ensemble head, loaded from checkpoints/ensemble_head.pt, scores these features. The returned {valid} is final_score >= threshold, so the verdict comes from learned weights, not hand-written rules.
✓no lean execution or reimplementationThe trusted no_lean audit passed with no violations, no Lean source files and nothing uninspected. Serving code (serve.s
The trusted no_lean audit passed with no violations, no Lean source files and nothing uninspected. Serving code (serve.sh, server.py) has no subprocess call, Lean toolchain or Lean checker reimplementation. Lean 4.15 was used only during development, to label semantic mutations in training/create_matched_corpus.py, and is not on the inference path.
✓stateless per requestserver.py handles each /check request on its own: one tokenized input, a forward pass and feature scoring. There is no c
server.py handles each /check request on its own: one tokenized input, a forward pass and feature scoring. There is no cross-request cache or memory; the cache is cleared every call. The head and models are fixed after loading.
✓actual inference uses reviewed weightsserve.sh runs server.py with BASE_MODELS_DIR, CKPT_ROOT_DIR and HEAD_PATH=checkpoints/ensemble_head.pt. server.py loads
serve.sh runs server.py with BASE_MODELS_DIR, CKPT_ROOT_DIR and HEAD_PATH=checkpoints/ensemble_head.pt. server.py loads the shipped base model shards under models/, merges the shipped LoRA adapters (checkpoints/ckpt_ext_*/lora/adapter_model.safetensors) and loads the ensemble head whose fold_models tensors appear in the weight inventory. Leftover head.pt and val_probs.npy files are not used by serving.
✓training data provenancetraining/data_manifest.json and architecture.json declare the training corpora: Goedel-LM/Lean-workbook-proofs, SFT_data
training/data_manifest.json and architecture.json declare the training corpora: Goedel-LM/Lean-workbook-proofs, SFT_dataset_v2 and RL_dataset_V2 (train splits, pinned revisions) plus the task's public validation.jsonl. training/download_h200.py and its log record the downloads (trajectory#92). create_matched_corpus.py keeps only lean_workbook_* ids. It uses miniF2F test/valid only as a decontamination blacklist (trajectory#123). No undeclared corpus was found.
✓non specialist initializationEvery base model is an openly released prover LM loaded at its declared revision: Goedel-Prover-V2-8B @dfd02e62, DeepSee
Every base model is an openly released prover LM loaded at its declared revision: Goedel-Prover-V2-8B @dfd02e62, DeepSeek-Prover-V2-7B and Goedel-Prover-SFT. The LoRA adapters (r=16) and the ridge head were trained during the session by training/pretrain_external_lora.py and export_head.py. adapter_config.json points at the declared local base. Neither the shipped files nor the trajectory show a pre-trained Lean-validity verifier checkpoint being used. Kimina-Prover-Preview-Distill-7B was downloaded but not shipped or used in serving.
✓harness credentials used only by the harnessThe trajectory shows only harness CLI use: spin compute, spin compute exec/upload, and spin openrouter models to list mo
The trajectory shows only harness CLI use: spin compute, spin compute exec/upload, and spin openrouter models to list models. The search found no API keys, no OpenRouter key creation and no LLM API calls (no openrouter.ai, anthropic or openai endpoint use). HF downloads used ordinary hub access. No credential misuse was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 7m into the runtook 15m 45s
startup 23.9 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 23.9 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 337.8 s plus 0.2 s for the first request, 10.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests