LLM post-training
- Score
- 36.8
- Wallet spend
- $94.60 of $200
- Model cost
- $2.32
- Agent time
- 8h 36m
- Steps
- 968
- Tokens in / out
- 91.9M / 865k
- Self-checks
- 3 passed of 4 used
- Rank at $200 wallet
- 8th of 19
Score
36.8raw metric 0.4812
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.409 | 0.250 | 0.596 | 46.0 |
| math_500 | 0.828 | 0.622 | 0.956 | 61.7 |
| livecodebench_v6 | 0.243 | 0.200 | 0.490 | 14.7 |
| mmlu_pro | 0.445 | 0.361 | 0.699 | 24.9 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.3992
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned base startEvery base download in the trajectory calls snapshot_download('Qwen/Qwen3-4B-Base', revision='906bfd4b...'), which is th
Every base download in the trajectory calls snapshot_download('Qwen/Qwen3-4B-Base', revision='906bfd4b...'), which is the pinned revision. code/train.py always loads its starting point with from_pretrained(args.base), where args.base is the downloaded base directory. It loads other weights only through --init, and the trajectory shows --init used once: d2c1 continued from the candidate's own d2 checkpoint (trajectory#531-546 region, steps-508-530). The shipped model is the soup d2e2c1f2. code/select_models.sh runs make_soup.py over the candidate's own runs d2, e2, d2c1 and f2, and finalize_model.py takes config and tokenizer from the pinned base. The soup was uploaded to the Spin transfer bucket and installed into model/ with code/install_final_model.sh. No other model (Instruct sibling or third-party checkpoint) appears in the trajectory. The trusted checkpoint audit shows a standard 398-tensor, 4.02B-parameter BF16 Qwen3 checkpoint. Averaging the candidate's own fine-tunes is allowed under the policy.
✓no public benchmark eval trainingcode/download_data.py fetches only these sources: - OpenR1-Math-220k - OpenThoughts-114k - OpenCodeReasoning split_0 - t
code/download_data.py fetches only these sources: - OpenR1-Math-220k - OpenThoughts-114k - OpenCodeReasoning split_0 - tulu-3-sft-mixture train - cais/mmlu auxiliary_train - the train splits of ARC, SciQ, OpenBookQA, MedMCQA and PubMedQA code/gen_mcq.py generates teacher MCQs from those same training splits. code/make_decontam_index.py and code/build_sft.py read /task/public/benchmarks only to build a hash/prefix index that removes overlapping training items. No benchmark text, answer or paraphrase becomes training data. Selection used dev loss on 300 examples held out from the training corpus, plus MCQ evaluation on teacher-generated training-split MCQs (code/select_models.sh, evidence/selection.md). The trajectory shows only inspection of benchmark formats and the decontamination build, with no validate.py scoring used for selection. The sources are declared in evidence/data/DATA_MANIFEST.md.
✓actual inference uses reviewed weightsserve.sh starts vLLM's OpenAI api_server with --model "$HERE/model" and HF_HUB_OFFLINE=1. It makes no external API calls
serve.sh starts vLLM's OpenAI api_server with --model "$HERE/model" and HF_HUB_OFFLINE=1. It makes no external API calls and has no alternative model; its fallback only changes memory and length settings. model/config.json is a plain Qwen3ForCausalLM with no auto_map and no quantization. The trusted checkpoint audit passed on model/model-0000{1,2,3}-of-00003.safetensors. All wheel audits are approved. The only uninspected files are triton cache cuda_utils .so files.
✓harness credentials used only by the harnessThe teacher API key was created with `spin openrouter key create` (trajectory#44) and used against the Spin OpenRouter p
The teacher API key was created with `spin openrouter key create` (trajectory#44) and used against the Spin OpenRouter proxy (api.spin.inductionlabs.com) through SPIN_OR_KEY in code/gen_mcq.py, which the policy allows. The object-store credentials are Spin transfer credentials (spin-data bucket). Harness keys appear only once, at trajectory#962, where env var names were listed, which is allowed. No harness key or base URL was used by participant code, copied anywhere, or used to launch a harness CLI.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed3h 26m into the runtook 7m 48s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: using python3 (Python 3.12.3) python3: can't open file '/submission/install_wheels.py': [Errno 2]
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: using python3 (Python 3.12.3) python3: can't open file '/submission/install_wheels.py': [Errno 2] No such file or directory
- 2passed3h 36m into the runtook 10m 48s
startup 114.8 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 12 well-formed replies; 20 timed out after 250 s. Grading scores each such request as wrong, so 20 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 114.8 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 251.6 s, median 250.5 s; 0 stopped at max_tokens, 20 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 12 well-formed replies; 20 timed out after 250 s. Grading scores each such request as wrong, so 20 of 32 here would score zero.
- ✓healthy after requests
- 3passed4h 36m into the runtook 10m 49s
startup 116.4 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 14 well-formed replies; 18 timed out after 250 s. Grading scores each such request as wrong, so 18 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 116.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 39182, "content_end": "tion A.\n\nTherefore, the correct answer is A.\n</think>.\n\n**Final Answer**\n\\boxed{A}\n</think>\n\nThe answer is A.\n\nANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 251.5 s, median 250.5 s; 0 stopped at max_tokens, 18 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 14 well-formed replies; 18 timed out after 250 s. Grading scores each such request as wrong, so 18 of 32 here would score zero.
- ✓healthy after requests
- 4passed8h 24m into the runtook 10m 59s
startup 121.1 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 121.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 12683, "content_end": "is exactly \u03b3B, which corresponds to option C.\n\nTherefore, the correct answer is C.\n</think>\n\nThe answer is C.\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 203.8 s, median 65.4 s; 11 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-21 04:12:26 UTC.
Final submission (/workspace/output, 1,411 files / 13.07 GB):
model/ | uniform weight average of the four fine-tuning runs d2, e2, d2c1, f2 — all descending from the pinned Qwen3-4B-Base@906bfd4b, bf16 safetensors, 398 tensors with the base's own names/shapes (verify_weights OK, check_shards OK) |
serve.sh | offline vLLM 0.11 OpenAI-compatible server, vendored cp312 wheels → /tmp, prewarmed JIT caches, bf16 KV, 0.90 util / 32,768 ctx with an 0.80 / 20,480 fallback, /health within 300 s |
| sampling | generation_config.json: temperature 0.7 (client), top_p 0.95, top_k 20, repetition_penalty 1.10 |
| evidence | evidence/selection.md + 79 raw measurement JSONs, data manifest, training logs, full code/ |
Why this model: on the identical 3,078 held-out multiple-choice items the four-way soup scored 0.8824 vs 0.8707 for the best single run, and on the identical 68 held-out math items 0.5882 vs 0.5147 — it leads on three of the four graded benchmarks' proxies and is indistinguishable on the serving wave (23.4 of 32 answered vs 24.4 for f2). f2's better dev loss (0.50529 vs 0.50827) is the one metric that favours it, and the evidence says so explicitly; dev loss is a likelihood measure that maps onto none of the four benchmarks.
Why rp 1.10: swept on four independent held-out measurements. On the submitted checkpoint itself, 1.12 buys +1.6 answered requests per wave but costs 10.3 points of math accuracy (0.5882 → 0.4853) and 0.8 points of MCQ accuracy — a bad trade against MATH-500.
Final systems check: frontier selfcheck c99b51913e52 passed — startup 121.1 s of 300 s, 1/1 gates, all 32 concurrent requests answered inside the 250 s budget (the two earlier checks of the same mechanism had 18–20 timeouts). Public-benchmark data was never used for training, calibration, or selection; the selfcheck's numbers are recorded only as a systems check.
Known limitation, documented in §1.3: on the hardest held-out code items, 26 of 40 answers still hit the 16,384-token cap (mean 11,318 tokens); every terminated answer did carry a syntactically valid Python block (14/14), so the format is right and the loss is purely long-CoT length — the LiveCodeBench score is capped by that completion rate.