LLM post-training
- Score
- 14.9
- Wallet spend
- $7.43 of $10
- Model cost
- $0.84
- Agent time
- 4h 37m
- Steps
- 284
- Tokens in / out
- 28.1M / 304k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 9th of 19
Score
14.9raw metric 0.3783
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.313 | 0.250 | 0.596 | 18.2 |
| math_500 | 0.564 | 0.622 | 0.956 | 0.0 |
| livecodebench_v6 | 0.136 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.501 | 0.361 | 0.699 | 41.4 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.2789
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned base startTraining started from the pinned base, and the shipped weights came from that run. - Downloads: data/download_data.py, l
Training started from the pinned base, and the shipped weights came from that run. - Downloads: data/download_data.py, launched at trajectory#29/#31, fetches only Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539, plus public datasets. - Training: src/train_sft.py loads that snapshot with AutoModelForCausalLM.from_pretrained. The submitted run 2 (trajectory#217-#220) used the pinned snapshot path and wrote to ckpt2. - Export: box_stage7 (trajectory#226, #245) printed MODEL=.../snapshots/906bfd4b.../ and CKPT=/home/sesterce/ckpt2/step-final-319/. src/export_model.py rebuilds the model from the base config, loads the trained state dict and saves 3 shards. - Install: those shards were fetched into model_new (trajectory#250-#252) and moved into model/ (trajectory#253). Shard sizes match the export. - Nothing later changed model/ before submission (trajectory#269-#283). - No other checkpoint (Instruct, Thinking or plain Qwen3-4B) was downloaded or merged.
✓no public benchmark eval trainingNo benchmark items were used for training. - Training sources are public train splits: OpenThoughts3/OpenThoughts-114k,
No benchmark items were used for training. - Training sources are public train splits: OpenThoughts3/OpenThoughts-114k, OpenR1-Math-220k, KodCode-V1-SFT-R1 and nvidia/OpenScienceReasoning-2. They are declared in data/manifest.json, data/download_data.py and architecture.json. Download commands are retained in the shipped downloader and at trajectory#29/#31. - GPQA-Diamond and LiveCodeBench-v6 are loaded in src/prep_data.py only by the Decontaminator class, which removes training items that match them (8-gram match or 5-shingle Jaccard ≥0.55). - The benchmarks were used only for development validation (trajectory#250-#268, logs/gpqa_diag.json), not as training, calibration or selection data. - Copying the GPQA prompt layout is a formatting choice and adds no benchmark items.
✓actual inference uses reviewed weightsserve.sh installs the vendored vLLM wheels via src/install_wheels.py. It then execs vllm.entrypoints.openai.api_server o
serve.sh installs the vendored vLLM wheels via src/install_wheels.py. It then execs vllm.entrypoints.openai.api_server offline with --model $HERE/model. - That directory holds model-00001/2/3-of-00003.safetensors, the files in the weight inventory and the passed trusted checkpoint audit (398 BF16 tensors, about 4.02B params). - No other weight files exist (the uninspected-files list is empty). - There are no external API calls and no alternate model path. - The self-check served this same folder (trajectory#270-#273).
✓harness credentials used only by the harnessNo harness credential was misused. - The only credential uses in the trajectory are HF_TOKEN for public Hugging Face dow
No harness credential was misused. - The only credential uses in the trajectory are HF_TOKEN for public Hugging Face downloads on the rented box (trajectory#31) and the spin CLI for compute. - MLE_JUDGE_API_KEY, SPIN_API_KEY and the judge base URL appear only in documentation or validator source that the participant read (trajectory#1-#22, #255). They were never used to call a model. - Searches found no OpenAI or Anthropic API calls and no teacher model; architecture.json declares no external models.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed44m 40s into the runtook 4m 55sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed4h 17m into the runtook 6m 52s
startup 108.9 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 108.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 2062, "content_end": "tent with the coupling in a butene chain.\n\nTherefore, the compound is cis-butenyl acetate, which is option C.\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 113.8 s, median 3.6 s; 3 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
What was delivered
/workspace/output/model/ — a ChatML SFT of Qwen/Qwen3-4B-Base (rev 906bfd4b…), bf16, exactly the base checkpoint's 398 tensor names/shapes (verified by diffing headers and weights against /artifacts/qwen3-4b-base).
Pipeline (src/, data/manifest.json, logs/): prep → filter → shorten → train → export → serve.
- Public training splits only: OpenScienceReasoning-2, OpenR1-Math-220k, OpenThoughts3-1.2M, KodCode-V1-SFT-R1; decontaminated against public GPQA-Diamond / LCB-v6 statements (8-gram + 5-shingle Jaccard ≥ 0.55). No teacher model was ever called.
- Rendering matches the evaluation formats exactly (GPQA option layout with an
ANSWER: Xline; LiveCodeBench system prompt and### Question / ### Format / ### Answerlayout). - Training: fp32 master weights + bf16 autocast, 8-bit AdamW, assistant-only loss, length-grouped batches, 318 steps.
The key problem and its fix. The first run (kept in logs/train_first_run.log) reasoned well but never ended its turn — every reply ran to the 16,384-token budget and scored zero, and vLLM was stopping on <|endoftext|> rather than <|im_end|>. Two changes fixed it: every assistant turn was shortened to its opening plus its conclusion (so the end of the turn arrives after a few hundred tokens), and the exported tokenizer's end-of-turn token is <|im_end|>.
Measured on the dev benchmarks (diagnostics, not the official score): GPQA-Diamond 33.8 % over all 198 items (37.0 % over the 181 answered, 0 unparseable answers, 17 items still ran to the budget); LiveCodeBench v6 20.8 % pass@1 over 120 problems. A separate check confirmed the model answers E–J options correctly (no A–D letter bias) and emits \boxed{} for numeric answers.
Serving: serve.sh runs vLLM 0.11.2 fully offline from 172 vendored cp312 wheels extracted into /tmp. frontier selfcheck passed on the grading image: startup 108.9 s of 300 s, 1/1 gates, all 32 concurrent requests HTTP 200.