Lean proof validity
- Score
- 23.4
- Wallet spend
- $6.25 of $10
- Model cost
- $21.84
- Agent time
- 3h 5m
- Steps
- 374
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 8th of 19
Score
23.4raw metric 0.6172
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.sh runs src/server.py. For each request, InferenceEngine._run_batch does a forward pass of the merged Kimina causa
serve.sh runs src/server.py. For each request, InferenceEngine._run_batch does a forward pass of the merged Kimina causal LM backbone (last 4 hidden states plus LM surprise statistics). It then runs 6 learned ClassificationHeadModule heads from src/model.py, combines them as a weighted ensemble and compares the result with a calibrated threshold. The 27 regex structural features are only inputs to the trainable heads (stats_direct is a learned parameter). They do not decide the verdict.
✓no lean execution or reimplementationThe trusted no-Lean audit passed with no violations and nothing left uninspected. src/server.py has no subprocess, Lean
The trusted no-Lean audit passed with no violations and nothing left uninspected. src/server.py has no subprocess, Lean or network calls. src/verify_with_lean.py compiled perturbed external samples with Lean offline to label training data (trajectory#118). It is not imported or called by the serving path. Using Lean offline to label training data is not inference-time Lean execution.
✓stateless per requestsrc/server.py has a maximum batch size of 1 and keeps no cache or memory across requests. Each request is tokenized and
src/server.py has a maximum batch size of 1 and keeps no cache or memory across requests. Each request is tokenized and scored on its own using fixed loaded weights.
✓actual inference uses reviewed weightsThe server loads model/heads_and_config.pt (6 head state dicts, head weights and threshold) and AutoModelForCausalLM fro
The server loads model/heads_and_config.pt (6 head state dicts, head weights and threshold) and AutoModelForCausalLM from model/backbone (model/backbone/model.safetensors), with HF_HUB_OFFLINE=1. The trajectory shows the trained kimina_v3 artifacts copied into /workspace/output/model (trajectory#349, #351). The threshold was then edited to 0.648 (trajectory#352). The final self-check server log shows these files loading (trajectory#372). The weight inventory for heads_and_config.pt matches the head shapes in model.py.
✓training data provenanceThe training corpora are declared in architecture.json and data_manifest.json, with download code in src/download_data.p
The training corpora are declared in architecture.json and data_manifest.json, with download code in src/download_data.py and logs in logs/data_download.log and logs/dataset_build.log. They are Goedel/DeepSeek-Prover train splits, SFT_dataset_v2 pairs, regex perturbations and Lean-labelled perturbations of those train rows, plus the allowed validation.jsonl (5-fold GroupKFold CV, then threshold calibration). miniF2F, ProverBench, MathOlympiadBench and the Goedel minif2f/proofnet/putnam files were downloaded only to build a decontamination denylist in src/build_dataset.py. The trajectory shows them only being inspected and used for filtering (trajectory#53), never for training or selection.
✓non specialist initializationsrc/train.py loads AutoModelForCausalLM.from_pretrained(args.base_model, revision=...). The final run was launched with
src/train.py loads AutoModelForCausalLM.from_pretrained(args.base_model, revision=...). The final run was launched with --base-model AI-MO/Kimina-Prover-Distill-1.7B at revision 3d66bc56... (trajectory#305), which matches architecture.json. Kimina is an allowed prover base, not a validity classifier, critic or reward model. LoRA was injected, fine-tuned and merged, and the heads were trained from scratch.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENAI, API_KEY, BASE_URL and GEMINI found no participant command that uses harness m
Trajectory searches for ANTHROPIC, OPENAI, API_KEY, BASE_URL and GEMINI found no participant command that uses harness model credentials. The Gemini 429 quota messages are harness system messages (e.g. trajectory#218), not participant API calls. HF_TOKEN was used only to query Hugging Face for downloads (steps-28-49), which is allowed. The S3 session keys were used only for spin compute file transfer.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 12m into the runtook 6m 21s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [Server] Loading tokenizer and config from /submission/model... [Server] Loading merged CausalLM b
3 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [Server] Loading tokenizer and config from /submission/model... [Server] Loading merged CausalLM backbone from /submission/model/backbone... `torch_dtype` is deprecated! Use `dtype` instead! Traceback (most recent call last): File "/submission/src/server.py", line 219, in <module> main() File "/submission/src/server.py", line 212, in main ENGINE = InferenceEngine(Path(args.model_dir), device="cuda", max_batch_size=args.max_batch_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/src/server.py", line 57, in __init__ backbone = AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4806, in from_pretrained raise ValueError( ValueError: Using a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate`. You can install it with `pip install accelerate`
- 2passed2h 22m into the runtook 3m 58s
startup 6.2 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 150.6 s plus 0.1 s for the first request, 23.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed2h 58m into the runtook 5m 58s
startup 7.0 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 7.0 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 270.0 s plus 0.1 s for the first request, 13.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests