Lean proof validity
- Score
- 45.3
- Wallet spend
- $35.49 of $50
- Model cost
- $31.22
- Agent time
- 6h 38m
- Steps
- 415
- Tokens in / out
- 123.1M / 226k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 1st of 14
Score
45.3raw metric 0.7265
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passserve.py loads AutoTokenizer and AutoModelForSequenceClassification (Qwen3ForSequenceClassification, 36 layers, hidden s
serve.py loads AutoTokenizer and AutoModelForSequenceClassification (Qwen3ForSequenceClassification, 36 layers, hidden size 4096) from the shipped model directory onto cuda in bf16. run_member does a padded forward pass and takes the single score logit. The verdict is that logit > threshold. The weight inventories hold the full Qwen3-8B tensors plus the trained score.weight [1,4096] in shard 4.
✓no lean execution or reimplementationThe serving path (serve.sh -> serve.py) only strips the header, tokenizes, truncates, runs the model and compares to a t
The serving path (serve.sh -> serve.py) only strips the header, tokenizes, truncates, runs the model and compares to a threshold. There is no subprocess, no Lean binary, no network call and no rule-based checking of Lean. The trusted no_lean audit passed with no violations, no review leads and 0 Lean source files. Lean v4.15.0 was used only offline during development to label training data.
✓stateless per requestencode() runs per request. gpu_loop groups up to 256 requests within a 10 ms window purely for throughput, and each item
encode() runs per request. gpu_loop groups up to 256 requests within a 10 ms window purely for throughput, and each item gets its own padded sequence and its own logit. No cache, memory or cross-request state changes any verdict. The threshold is fixed and loaded from model/threshold.json at startup.
✓actual inference uses reviewed weightsserve.sh runs serve.py offline. serve.py loads the shipped model/ directory: model/ensemble.json is absent, so it falls
serve.sh runs serve.py offline. serve.py loads the shipped model/ directory: model/ensemble.json is absent, so it falls back to the single member using model/threshold.json. The config and the four safetensors shards match the reviewed inventories. The trajectory shows runs/final/model was downloaded to models_out/final with matching MD5s (trajectory#376). It was then copied into /workspace/output/model together with threshold_final.json (trajectory#377). No later step replaced it. The threshold's val_auc of 0.8176 matches the final training eval and train/logs/calibration_final.txt.
✓training data provenanceDATA_MANIFEST.md, data_scripts/download_public.py and download_public.log declare and download pinned public HF corpora.
DATA_MANIFEST.md, data_scripts/download_public.py and download_public.log declare and download pinned public HF corpora. Generated attempts came from public prover models (DeepSeek-Prover-V1.5-RL, Goedel-Prover-SFT, STP, DeepSeek-Prover-V2-7B), plus mutations, and were labeled by the participant's own Lean compilation. data_scripts/build_pool.py and its log use miniF2F test/valid and the validation statements only as a decontamination reference (TF-IDF threshold 0.6). The shipped validation.jsonl was used for evaluation, calibration and an abandoned fine-tuning experiment, all of which the policy allows. I found no prohibited corpus being downloaded or used.
✓non specialist initializationarchitecture.json declares the base as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. This is a prover base, which
architecture.json declares the base as Goedel-LM/Goedel-Prover-V2-8B at a pinned revision. This is a prover base, which the policy allows. The final run (trajectory/steps-236-270, step 264) is `train_cls.py --model ~/models/Goedel-Prover-V2-8B ... --out runs/final`, and its log shows score.weight newly initialized. train/train_cls.py applies LoRA, merges it and saves the result. The fine-tune and second-epoch experiments that started from earlier runs were not shipped.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENAI, api_key and BASE_URL found no participant use of harness credentials. The onl
Trajectory searches for ANTHROPIC, OPENAI, api_key and BASE_URL found no participant use of harness credentials. The only match is SPIN_BASE_URL in the compute-tool documentation. The shipped code uses only HF downloads and runs the server offline.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 17m into the runtook 5m 44s
startup 9.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.3 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 135.7 s plus 0.2 s for the first request, 26.1 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed5h 44m into the runtook 6m 6s
startup 9.7 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 9.7 s
- ✓POST /check · {"id": "v00000", "valid": false}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 134.3 s plus 0.1 s for the first request, 26.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 10:58 UTC; it is now being graded.