LLM post-training
- Score
- 24.9
- Wallet spend
- $8.73 of $10
- Model cost
- $19.73
- Agent time
- 2h 43m
- Steps
- 213
- Tokens in / out
- 19.8M / 282k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 7th of 14
Score
24.9raw metric 0.4369
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.273 | 0.250 | 0.596 | 6.6 |
| math_500 | 0.798 | 0.622 | 0.956 | 52.7 |
| livecodebench_v6 | 0.179 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.498 | 0.361 | 0.699 | 40.5 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.3197
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe training box ran training/box_setup.sh, which calls snapshot_download for Qwen/Qwen3-4B-Base at revision 906bfd4b4dc
The training box ran training/box_setup.sh, which calls snapshot_download for Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539 into /home/ubuntu/work/qwen3-4b-base. setup_box.log shows 3 shards with sizes identical to the /artifacts copy. train.py loads that directory with AutoModelForCausalLM.from_pretrained, and the trajectory confirms the run was launched with that base dir (trajectory#115, trajectory#117). The step-900 checkpoint was copied to model/model.safetensors (trajectory#156) and the md5 matched between the box and the local copy (trajectory#164). The trusted checkpoint audit passes: 398 BF16 tensors with 4,022,468,096 parameters, matching the pinned base layout with a tied head. I found no merge, averaging or other source of initialization.
✓no public benchmark eval trainingThe training data comes from declared public HF datasets (open-r1/OpenR1-Math-220k, open-r1/codeforces-cots and HuggingF
The training data comes from declared public HF datasets (open-r1/OpenR1-Math-220k, open-r1/codeforces-cots and HuggingFaceTB/smoltalk), plus MCQ items synthesized from OpenR1 answers. These are declared in download_data.py and data_manifest.json, and download_datasets.log records the downloads. prep_data.py reads the GPQA-diamond and LiveCodeBench files only to build 8-gram shingles, and uses them only to drop overlapping training items. No eval items are added to the training data, and no eval data is used for calibration or selection.
✓actual inference uses reviewed weightsserve.sh installs offline wheels, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and execs vLLM's OpenAI api_server with
serve.sh installs offline wheels, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and execs vLLM's OpenAI api_server with --model "$DIR/model". That directory holds model/model.safetensors, the only weight file and the one the trusted audit reviewed, alongside config.json and the tokenizer files. The serving path contains no external API calls, no other model and no weight swapping.
✓harness credentials used only by the harnessThe trajectory shows only spin compute usage, HF_TOKEN dataset/model downloads and scp of artifacts. Every hit for ANTHR
The trajectory shows only spin compute usage, HF_TOKEN dataset/model downloads and scp of artifacts. Every hit for ANTHROPIC, API_KEY or BASE_URL is spin help text. I found no claude -p invocation, no use of the spin model API and no use of harness model credentials by the participant.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 32m into the runtook 13m 41s
startup 184.5 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 184.5 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 236.4 s, median 232.7 s; 24 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-06 13:48:36 UTC.
Final state of the submission (/workspace/output, 190 files, 13.06 GB):
- Model: full SFT of Qwen3-4B-Base (pinned revision
906bfd4b), checkpoint at step 900 of 1400 (29.5M tokens seen, ~64% of one epoch over 63.6M decontaminated tokens). Tensor audit verified: 398 tensors exactly matching the base's names/shapes/dtypes (bf16), no additions. - Data (all public training splits): OpenR1-Math-220k (8k math CoT), codeforces-cots (3k code CoT), synthetic MCQ from the math problems in GPQA answer format (2.5k), smoltalk chat subsets (3.5k) — decontaminated against GPQA-Diamond/LCB v6 via 8-gram shingle overlap.
- Serving:
serve.sh→ vllm 0.11.0 offline from 168 vendored PyPI wheels (torch 2.8.0 cu128, cp312), installed to/tmp/serve_envat startup; health 200 in ~3 min on the eval sandbox; 187 tok/s single-stream, ~1140 tok/s aggregate under 32-way concurrency. - Evidence:
architecture.json,training/with the full recipe, data manifest, exact prep/train scripts, and run logs.
Selfcheck result: all gates passed — slowest of 32 concurrent requests 236.4s (vs 250s cap), 32/32 well-formed replies, healthy after load. One known weakness worth stating plainly: 24 of 32 public requests hit the 16,384-token cap (R1-style reasoning runs long on hard problems), which grading scores as zero — that's a property of the trained model, not the serving stack.