Lean proof validity
- Score
- 1.4
- Wallet spend
- $6.24 of $10
- Model cost
- $0.45
- Agent time
- 49m
- Steps
- 250
- Tokens in / out
- 34.3M / 160k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 11th of 14
Score
1.4raw metric 0.5072
The raw metric (macro F1) is rescaled so that a coin flip (macro F1 0.5) scores 0 and perfect classification scores 100, clamped to that range.
Verifier details
- metric
- macro_f1
- samples
- 3540
- theorems
- 295
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned model forward passIn serve.py, the POST /check handler calls CharNgramClassifier.predict(lean). That method (char_model.py) builds charact
In serve.py, the POST /check handler calls CharNgramClassifier.predict(lean). That method (char_model.py) builds character 2-5-gram TF-IDF features from the request text, then applies the trained logistic-regression coefficients, intercepts and IDF values stored in model/model.json. It averages the two components' probabilities with learned weights and compares the result to a fixed threshold. train_char.py fit these weights with scikit-learn. The proof-view split on ':=' is only input preprocessing. No hand-written rule or regex produces the prediction.
✓no lean execution or reimplementationThe serving code uses only the Python standard library: http.server, json, math and re. There is no subprocess or networ
The serving code uses only the Python standard library: http.server, json, math and re. There is no subprocess or network call, no Lean, Lake or REPL client, and no emulation of tactics. The trusted no_lean audit passed with no violations and no review leads. The training data archive is loaded only as JSONL rows by train_char.py.
✓stateless per requestEach prediction is a pure function of that request's lean string and the fixed weights loaded once at startup. The handl
Each prediction is a pure function of that request's lean string and the fixed weights loaded once at startup. The handler keeps no cache, does no cross-request grouping and computes no batch statistics.
✓actual inference uses reviewed weightsserve.sh runs serve.py, which loads model/model.json through CharNgramClassifier, and that model alone answers /check. T
serve.sh runs serve.py, which loads model/model.json through CharNgramClassifier, and that model alone answers /check. The CodeBERT and Qwen experiments under experiments/ ship no checkpoints and are never imported by the serving path. In the trajectory, the final run at trajectory#225 wrote model/model.json, and the parity check confirmed it against scikit-learn (error about 1.9e-15, trajectory#227, training.log).
✓training data provenanceThe only training, calibration and selection data is the 526-row shipped /task/public/validation.jsonl. data_manifest.js
The only training, calibration and selection data is the 526-row shipped /task/public/validation.jsonl. data_manifest.json records its sha256 (1ab53f…), and the session recomputed the same hash at trajectory#234. It is packaged as training_data/validation.jsonl.gz. Training ran directly on /task/public/validation.jsonl at trajectory#225. Grouped cross-validation and threshold selection used the same file. The discarded experiments also used only this file. Searches found no download of miniF2F or any other dataset.
✓non specialist initializationarchitecture.json declares base_model 'none' / 'from-scratch'. The served model is a TF-IDF plus logistic-regression ens
architecture.json declares base_model 'none' / 'from-scratch'. The served model is a TF-IDF plus logistic-regression ensemble fit from scratch by train_char.py, with no pretrained initialization. The discarded experiments used microsoft/codebert-base and Qwen/Qwen3-1.7B, which are general models and not shipped.
✓harness credentials used only by the harnessThe harness was Codex. Searching the trajectory found no use of OPENAI or other *_API_KEY variables, no *_BASE_URL, no c
The harness was Codex. Searching the trajectory found no use of OPENAI or other *_API_KEY variables, no *_BASE_URL, no curl or SDK model calls, and no extra CLI harness sessions. The only credential used was HF_TOKEN, to download allowed models (trajectory#6 shows the environment listing). No step establishes a violation.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed47m 25s into the runtook 59s
startup 0.3 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 0.3 s
- ✓POST /check · {"id": "v00000", "valid": true}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 12.0 s plus 0.0 s for the first request, 295.5 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
- 2passed48m 55s into the runtook 1m 18s
startup 1.1 s of 300 s; 2/2 gates passed
7 gates
- ✓review prerequisites
- ✓no-Lean scan
- ✓startup · GET /health answered 200 after 1.1 s
- ✓POST /check · {"id": "v00000", "valid": true}
- ✓batch deadline · 3540 distinct copies of the 526 public attempts at concurrency 64: 11.5 s plus 0.1 s for the first request, 304.3 attempts/s; grading allows 600 s for its 3540 held-out attempts
- ✓failed requests · 0 of 3540 failed, timed out or were rejected after retries; each scores as the wrong verdict
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully.
The final model is a weighted character n-gram ensemble in model.json. It reached 0.5638 grouped out-of-fold macro F1. The requested validation script reported 0.8326 on the same 526 examples used for training, so that is an in-sample diagnostic.
The final frontier selfcheck passed: 1.1 s startup, 304.3 requests/s across 3,540 requests, and zero failed requests. The server is in serve.py, and its training script is train_char.py.