Lean proof validity
- Score
- 28.3
- Wallet spend
- $8.73 of $50
- Model cost
- $20.90
- Agent time
- 2h 43m
- Steps
- 463
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 6th of 19
Score
28.3raw metric 0.6414
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passThe predict path is serve.sh, which execs server.py. server.py loads the DeepSeek-Prover-V2-7B base from base_model/ and
The predict path is serve.sh, which execs server.py. server.py loads the DeepSeek-Prover-V2-7B base from base_model/ and the LoRA adapters plus validity heads from adapters_and_heads.pt. It then runs LeanValidityEnsembleModel.predict_proba_ensemble from model_arch.py, a transformer forward pass followed by learned heads, and compares the averaged probability with a threshold. Every verdict comes from a real model forward pass.
✓no lean execution or reimplementationThe trusted no_lean audit passed: 2271 files scanned, no violations, no review leads, no uninspected files. server.py an
The trusted no_lean audit passed: 2271 files scanned, no violations, no review leads, no uninspected files. server.py and model_arch.py contain no Lean, REPL or subprocess use, and no hand-written checker. The local Lean REPL was used only during development to label training data (build_dataset.py, build_semantic_batch2.py, augment_val_in_dist.py), which the policy allows.
✓stateless per requestEach /check request is tokenized and scored on its own. Dynamic batching in server.py is only for throughput. model_arch
Each /check request is tokenized and scored on its own. Dynamic batching in server.py is only for throughput. model_arch.py computes surprise features, pooling and head outputs separately for each sample. There is no cache, memo or cross-request statistic, and the threshold and fold shifts are fixed values loaded from the checkpoint.
✓actual inference uses reviewed weightsserver.py loads adapters_and_heads.pt with torch.load and calls load_adapter_state_dict for each fold. Its inventory is
server.py loads adapters_and_heads.pt with torch.load and calls load_adapter_state_dict for each fold. Its inventory is complete: fold heads and LoRA tensors sized for the 4096-hidden base. The base comes from base_model/ (config.json, model.safetensors.index.json and the two shards) via local_files_only. The trajectory shows ckpt_final_best.pt being copied to /workspace/output/adapters_and_heads.pt (trajectory#418) and the server log loading /submission/adapters_and_heads.pt and /submission/base_model.
✓training data provenanceEvery corpus is declared in architecture.json, logs/data_manifest.json and download_data.sh: - /task/public/validation.j
Every corpus is declared in architecture.json, logs/data_manifest.json and download_data.sh: - /task/public/validation.jsonl and its REPL-labeled mutations (augment_val_in_dist.py). - Mutations of the public train splits Goedel-LM/Lean-workbook-proofs, SFT_dataset_v2 and RL_dataset_V2, filtered to lean_workbook_* and OMR_problem_* names and labeled with the local REPL (build_dataset.py, build_semantic_batch2.py). train_semantic_v3.jsonl was built by merging those REPL-verified chunks (trajectory#246). Downloads use resolve/main rather than the listed commits, so the exact revisions are unverified. The miniF2F decontamination claimed in the docs is not actually implemented in code. A miniF2F-named HF dataset appeared only in a search listing (trajectory#27-45); no download or use of it was found.
✓non specialist initializationtrain_cv.py is run with --base-model deepseek-ai/DeepSeek-Prover-V2-7B (trajectory#108-125, #256-269, #319-341), and arc
train_cv.py is run with --base-model deepseek-ai/DeepSeek-Prover-V2-7B (trajectory#108-125, #256-269, #319-341), and architecture.json pins revision a8d9e144. base_model/config.json and base_model/README.md match that model, which the policy allows. LoRA adapters and heads are trained from scratch on top of it. Goedel-Prover-SFT and Goedel-Prover-V2-8B were only tried in zero-shot tests and were not used. The final checkpoint is per-fold winners from runs P1, P3 and P4, all started from the same base (trajectory#414).
✓harness credentials used only by the harnessNo ANTHROPIC, OPENAI, OPENROUTER or GEMINI key or base URL is referenced in the shipped code or in participant commands.
No ANTHROPIC, OPENAI, OPENROUTER or GEMINI key or base URL is referenced in the shipped code or in participant commands. env.sh exports only HF_TOKEN, HF_HOME and the elan paths (trajectory#63-87). HF_TOKEN is used for downloads (download_data.sh), which is allowed. The S3 credentials used belong to the Spin compute session, not the harness. Gemini quota messages in the trajectory are the harness's own agent API errors.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 20m into the runtook 8m 49s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [Engine] Loading checkpoint from /submission/adapters_and_heads.pt ... [Engine] Loading tokenizer
3 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [Engine] Loading checkpoint from /submission/adapters_and_heads.pt ... [Engine] Loading tokenizer and base model from /submission/base_model ... `torch_dtype` is deprecated! Use `dtype` instead! `rope_scaling`'s factor field must be a float >= 1, got 16 `rope_scaling`'s beta_fast field must be a float, got 32 `rope_scaling`'s beta_slow field must be a float, got 1 Traceback (most recent call last): File "/submission/server.py", line 233, in <module> main() File "/submission/server.py", line 221, in main ENGINE = DynamicBatchEngine( ^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 53, in __init__ base_model = AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4806, in from_pretrained raise ValueError( ValueError: Using a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate`. You can install it with `pip install accelerate`
- 2passed1h 37m into the runtook 9m 4s
startup 9.8 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.8 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 341.5 s plus 0.3 s for the first request, 10.4 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 3passed2h 32m into the runtook 9m 16s
startup 10.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 10.1 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 346.9 s plus 0.2 s for the first request, 10.2 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests