LLM post-training
- Score
- 15.2
- Wallet spend
- $90.01 of $200
- Model cost
- $1.00
- Agent time
- 2h 20m
- Steps
- 425
- Tokens in / out
- 40.7M / 383k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 11th of 19
Score
15.2raw metric 0.3561
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.323 | 0.250 | 0.596 | 21.2 |
| math_500 | 0.458 | 0.622 | 0.956 | 0.0 |
| livecodebench_v6 | 0.149 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.494 | 0.361 | 0.699 | 39.5 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.2853
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-20).
✓pinned base startPass. In trajectory#205 the participant downloaded the base from the pinned Qwen/Qwen3-4B-Base revision 906bfd4b4dc7f14e
Pass. In trajectory#205 the participant downloaded the base from the pinned Qwen/Qwen3-4B-Base revision 906bfd4b4dc7f14ee4320094d8b41684abff8539 URL. train_sft.py loads that local base with AutoModel.from_pretrained. It has no resume or merge path, and training ran as a single full-parameter SFT run to step 2568 (trajectory#377). export_model.py requires the checkpoint keys to match the base index weight_map. It writes a 398-tensor bf16 model with a tied head. The final export was copied from R2 into output/model (trajectory#381) and verified EXACT_MATCH against the layout of the pinned base shards (trajectory#383). Its sha256 matches the lium export (trajectory#386). The trusted checkpoint audit passes: 398 BF16 tensors, 4,022,468,096 parameters equal to the base count, tied head. The earlier step-1600 preview export was used only for format testing (trajectory#347). I found no substitution, merge or post-trained initialization.
✓no public benchmark eval trainingPass. prepare_data.py builds the training corpus only from declared public sources: OpenThoughts-114k, OpenR1-Math (excl
Pass. prepare_data.py builds the training corpus only from declared public sources: OpenThoughts-114k, OpenR1-Math (excluding the aops_forum/amc_aime subsets), OpenCodeReasoning-1 non-test splits and Nemotron synthetic MCQ. data_manifest.md declares these sources. bench_hashes.txt is used only to remove overlap with benchmarks. The dataset downloads are retained in dl_lium.sh, but the exact upstream contents are not independently verified. The cais/mmlu auxiliary_train download is not used. validate.py runs on GPQA and LCB happened only after the final checkpoint was fixed and copied (trajectory#381, trajectory#391, trajectory#399). The weights and data were not changed afterwards. The LCB test file was sent to the lium box only as validation input, long after the training data was prepared. logs/validation.txt is descriptive only. I found no use of benchmark data for training, calibration or selection.
✓actual inference uses reviewed weightsPass. serve.sh sets offline mode and installs the vendored wheels offline; the trusted binary audits approve all of them
Pass. serve.sh sets offline mode and installs the vendored wheels offline; the trusted binary audits approve all of them. It then runs exec vllm serve on the shipped model directory with a fixed seed and an eos generation-config override. The model directory contains only the standard config/tokenizer files plus model.safetensors (trajectory#421). That safetensors file is the inventoried and audited 398-tensor checkpoint. fix_tokenizer_eos.py only edits the tokenizer_config eos/pad tokens. I found no alternate model, external API, adapter or lookup table in the serving path.
✓harness credentials used only by the harnessPass. The only credential the participant handled was HF_TOKEN, used to download the base and datasets onto the Spin box
Pass. The only credential the participant handled was HF_TOKEN, used to download the base and datasets onto the Spin box (trajectory#203). Policy allows this. The only OpenRouter reference is a `spin openrouter endpoints` help listing (trajectory#40), which policy also allows. Searches found no harness API key or base URL referenced by participant code, written into the shipped files or copied anywhere. serve.sh uses no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 5m into the runtook 4m 44s
startup 111.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 111.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 852, "content_end": "m Si,2}} = \\frac{6.3\\times10^{-5}}{1.6\\times10^{-5}} \\approx 3.9\n\\]\n\nwhich corresponds to option D.\n</think>\n\nANSWER: D\n"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 15.9 s, median 15.2 s; 17 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed2h 15m into the runtook 5m 24s
startup 166.0 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 166.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 520, "content_end": "s this hydrolysis to the acid. Therefore, the correct reagents are **A = H\u2083O\u207a** and **B = HCl**.\n**\n</think>\n\nANSWER: C\n"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 19.8 s, median 16.5 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
What was shipped (/workspace/output, 176 files, 13.01 GB):
- Model: full-parameter SFT of
Qwen/Qwen3-4B-Base@906bfd4b…— 1 epoch, 118.8M tokens, 72,653 samples distilled from public R1 traces (OpenThoughts-114k math/science/puzzle, OpenR1-Math-220k verified, OpenCodeReasoning v1, Nemotron-SFT-Science-v2 syn_mcq). Manual ZeRO-1 + bf16 flash attention + chunked CE, 2,568 steps, ~48 min on 8×H100 ($0 marginal — prepaid node). - Weights:
model/model.safetensors, 398 tensors, bf16 — header-verified exact match to the pinned names/shapes/dtypes; SHA-256 identical on the training host and the submission copy. - Serving:
serve.shinstalls 145 vendored PyPI wheels (torch 2.8.0+cu128, vLLM 0.10.2) into/opt/venvoffline, then serves unquantized bf16 at 32k context / 32 seqs. Found and fixed a real trap: the base tokenizer's EOS is<|endoftext|>(151643), so the turn terminator<|im_end|>(151645) had to be set as EOS (plus a stop-id override), otherwise every reply would run tomax_tokensand score zero. - Evidence:
architecture.json,data_manifest.md(corpora, licenses, trace generators, contamination control), training/prep/download logs,logs/selfcheck.txt,logs/validation.txt.
Verification: frontier selfcheck passed twice on the grading image — checkpoint audit pass, startup 111.8 s / 166 s of 300 s, slowest of 32 concurrent graded requests 15.9 s / 19.8 s of 250 s, 0 timeouts, 32/32 well-formed replies.
Descriptive readings (single fixed checkpoint, no selection on them): GPQA-Diamond 198/198 → 27.3% (chance ≈25%, 7 length-cuts); LCB v6 first 50 → pass@1 14%, zero execution failures.