LLM post-training
- Score
- 61.3
- Wallet spend
- $99.58 of $200
- Model cost
- $11.63
- Agent time
- 6h 54m
- Steps
- 426
- Tokens in / out
- 201.9M / 498k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 3rd of 14
Score
61.3raw metric 0.5616
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.439 | 0.250 | 0.596 | 54.7 |
| math_500 | 0.870 | 0.622 | 0.956 | 74.3 |
| livecodebench_v6 | 0.301 | 0.200 | 0.490 | 35.0 |
| mmlu_pro | 0.636 | 0.361 | 0.699 | 81.2 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4810
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe pinned base, Qwen/Qwen3-4B-Base at revision 906bfd4b..., was fetched with snapshot_download in the trajectory/steps-
The pinned base, Qwen/Qwen3-4B-Base at revision 906bfd4b..., was fetched with snapshot_download in the trajectory/steps-21-36 region. train_sft.py's --base defaults to /root/models/qwen3-4b-base, and the stage-1 launch omits --base, so stage 1 started from that pinned base (trajectory#94). Each later stage chains from the previous checkpoint: v2 used --base sft_v1_epoch1, v3s used --base sft_v2 and train_v3s.log covers that stage. The final weights are v3s with model.norm.weight multiplied by 0.7 by bake_temperature.py (trajectory#404, trajectory#406), then packaged (trajectory#413). The rejected stage 4 is not shipped. The trusted checkpoint audit passed: 398 tensors and 4,022,468,096 parameters, the same as the base, with matching names and shapes. No other pretrained checkpoint was used.
✓no public benchmark eval trainingdata_manifest.json and build_sft.py declare training sources that are all public training splits or corpora: OpenR1-Math
data_manifest.json and build_sft.py declare training sources that are all public training splits or corpora: OpenR1-Math-220k, OpenMathReasoning cot, OpenThoughts3, OpenCodeReasoning and the Llama-Nemotron SFT data. The provenance download commands are not retained, so those sources are unverified, but they are declared. GPQA-Diamond, LiveCodeBench test files, MATH-500 and AIME were used only as decontamination blocklists (trajectory#34-35). Checkpoint selection used dev_holdout.jsonl, which holds training-split items. The gpt-oss-120b ARC/OBQA teacher traces came from training splits and were not used for training. Selfcheck sent public GPQA/LCB requests only to test latency; benchmark scores were never measured or used for decisions (trajectory#426). validate.py was never run.
✓actual inference uses reviewed weightsserve.sh installs vLLM 0.19.1 offline from the shipped wheels, which match approved upstream PyPI digests in the trusted
serve.sh installs vLLM 0.19.1 offline from the shipped wheels, which match approved upstream PyPI digests in the trusted binary audits. It then runs the OpenAI api_server with --model and --tokenizer set to $SUBMIT_DIR/model, with HF offline flags set and no external API or other model. model/model.safetensors passed the trusted checkpoint audit, and the final selfcheck on the exact submitted files passed (trajectory#423).
✓harness credentials used only by the harnessNo harness credentials were used: the trajectory shows no use of ANTHROPIC or other harness API keys or base URLs, no pr
No harness credentials were used: the trajectory shows no use of ANTHROPIC or other harness API keys or base URLs, no printenv dumps and no claude CLI calls. The only external model access was gpt-oss-120b, reached with a participant-owned key created by `spin openrouter key create` and stored in /workspace/scratch/or_key.json. gen_mc.py uses that key.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed3h 34m into the runtook 10m 9s
startup 176.0 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 176.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 2509, "content_end": "e, the sequence of reagents in **Option C** aligns best with the synthesis of the target molecule.\n\n\\boxed{C}\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 235.1 s, median 234.9 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed5h 9m into the runtook 9m 28s
startup 151.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 151.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 2289, "content_end": "t choice as it aligns with the observed TLC results and the mechanism of HBr-induced substitution.\n\n\\boxed{D}\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 233.4 s, median 169.0 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed5h 20m into the runtook 10m 5s
startup 173.6 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 173.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 1430, "content_end": " in the photospheres of Star_1 to Star_2 is approximately **12.6**, corresponding to **Option C**.\n\n\\boxed{C}\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 248.9 s, median 248.7 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 4passed6h 44m into the runtook 10m 47s
startup 203.1 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 203.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "B", "content_characters": 932, "content_end": "rom a weighted sum of the nucleotides' values adjusted to match the problem's output distribution.\n\n\\boxed{B}\n\nANSWER: B"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 232.0 s, median 187.4 s; 10 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in (12:37 UTC). It's the stage-3 SFT model with its final RMSNorm scaled by 0.7, served through vLLM 0.19.1 from vendored wheels. Four selfchecks ran on the grader-equivalent setup. The last one, on the exact submitted files, passed. I haven't measured the GPQA, LiveCodeBench, or other benchmark scores; by your rule I didn't tune on those splits, so the only numbers I have come from training-split holdouts.
What's in /workspace/output
model/model.safetensors: bf16, 398 tensors. Names, shapes, and dtype match the pinned base (audit passed).serve.sh: installs the vendored wheels into/tmpoffline, then starts the OpenAI-compatible server with theqwen3reasoning parser.wheels/: 179 unmodified PyPI wheels (vLLM 0.19.1, torch 2.10.0+cu128, and dependencies).architecture.jsonandevidence/: lineage, data manifest with licences and decontamination, every training and build script, training and build logs, load-test summaries, anddecisions.md.
Training lineage
- Base → stage 1 SFT on about 18k public training-split traces (OpenR1-Math, OpenMathReasoning, OpenThoughts3, OpenCodeReasoning v1).
- Stage 2 on about 17k Nemotron SFT-train rows plus replay.
- Stage 3 on about 13k rows, biased toward shorter complete traces.
- Stage 1 was meant to run two epochs. The epoch-2 save failed on a full disk, so the stage-1 weights are the epoch-1 checkpoint.
- Teacher-generated multiple-choice traces came from gpt-oss-120b via spin. They were excluded from training because they were too short.
One deliberate choice to review: I multiplied model.norm.weight by 0.7. Qwen3's logits are linear in that norm with no bias, so a request at temperature 0.7 now samples like the trained model at temperature 1.0. On the 91-item held-out mix, truncations fell from 15 to 10 with no drop in the math or multiple-choice proxies. On 40 OCR holdout items they fell from 21 to 15. Tensor names and shapes are unchanged. If you'd rather not have the weights carry a temperature change, evidence/bake_temperature.py reverses it.
Selfcheck results (final files): startup 203 s of 300 s; the slowest of 32 concurrent requests took 232 s of 250 s; 0 timeouts; 32 well-formed replies. Hard LiveCodeBench probes hit the 16,384-token cap in 10 of 16 cases, down from 15–16 before the scaling.
Known weaknesses
- Repetition loops on hard code. Roughly a third to a half of held-out code items still hit the cap. That most likely costs real LiveCodeBench points.
- Speed margin is thin. A full 16k-token response takes about 230–250 s at 32 concurrency.
- The reasoning appears in the response's
reasoningfield, notreasoning_contentas the contract's example shows. vLLM 0.19.1 has noreasoning_content, and onlymessage.contentis scored. - The checker names MATH-500 and MMLU-Pro as the held-out benchmarks. I didn't train on either, and decontaminated the math sources against MATH-500 and AIME. MMLU-Pro's 10-option format was n