LLM post-training
- Score
- 49.3
- Wallet spend
- $46.00 of $50
- Model cost
- $39.87
- Agent time
- 9h 45m
- Steps
- 485
- Tokens in / out
- 47.9M / 429k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 2nd of 14
Score
49.3raw metric 0.5225
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.434 | 0.250 | 0.596 | 53.3 |
| math_500 | 0.834 | 0.622 | 0.956 | 63.5 |
| livecodebench_v6 | 0.271 | 0.200 | 0.490 | 24.5 |
| mmlu_pro | 0.551 | 0.361 | 0.699 | 56.1 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4441
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json declares Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539. In the session, the
architecture.json declares Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539. In the session, the base checkpoint was copied from /artifacts/qwen3-4b-base to S3, then staged to ~/base on the training box (trajectory#197). Stage 1 trained with --base /home/Ubuntu/base and started at a base-like loss of about 0.92 (trajectory#199). chain.sh continued from that output: out_final was used for stage 1b (out_1b) (trajectory#354). chain2.sh continued from out_1b for stage 2b (out_1b_s2b) (trajectory#433). The shipped model/ was copied from final_1b_s2b, and its hashes match the remote out_1b_s2b (trajectory#469, trajectory#470). The trusted checkpoint audit passed with 4,022,468,096 parameters and the same tensor count as the base. I found no download of any other checkpoint.
✓no public benchmark eval trainingevidence/data_provenance.json declares public train-split sources (OpenScienceReasoning-2, OpenR1, OpenCodeReasoning). I
evidence/data_provenance.json declares public train-split sources (OpenScienceReasoning-2, OpenR1, OpenCodeReasoning). It also records n-gram decontamination against GPQA-Diamond and LiveCodeBench v6. The benchmark files were used only to build the exclusion list, which build_devset.py loads through build_decontam(). Checkpoints were chosen on a self-built dev set made from unused, decontaminated train rows, plus a format probe (code/build_devset.py, code/fmt_test.py, logs/final_choice.json). I saw no training or selection on public benchmark eval items. The download logs for these sources were not reviewed, so their provenance is declared but unverified.
✓actual inference uses reviewed weightsserve.sh runs vLLM's OpenAI api_server with --model "$HERE/model", with HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 set.
serve.sh runs vLLM's OpenAI api_server with --model "$HERE/model", with HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 set. No other model or remote API is called. The optional ThinkBudget logits processor (code/think_budget.py) only reads tokenizer.json and forces close-thinking or end-of-turn tokens on the same model's logits. It makes no external calls and loads no other weights. The trusted inventory audit of model/model-0000{1,2}-of-00002.safetensors passed (398 BF16 tensors with the base parameter count). Those files are the weights copied from the final training run (trajectory#469).
✓harness credentials used only by the harnessSearches of the trajectory and shipped files found no use of harness credentials, no harness CLI invocations (such as cl
Searches of the trajectory and shipped files found no use of harness credentials, no harness CLI invocations (such as claude -p) and no Anthropic base-URL overrides. I also found no HF uploads (no push_to_hub, upload_folder or hf upload). Compute and transfers went through Spin (spin compute, spin s5cmd), which is allowed (trajectory#197, trajectory#354). evidence/data_provenance.json says no model API was used to generate data.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed4h 30m into the runtook 9m 31s
startup 121.5 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 121.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 1273, "reasoning_characters": 53025, "content_end": "carboxylate \n- **B**: 3-(2-hydroxycyclohex-1-en-1-yl)butanenitrile \n- **C**: 2-hydroxycyclohexane-1,3-dione\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 185.0 s, median 176.6 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed7h 14m into the runtook 9m 3s
startup 105.0 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 105.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 1086, "reasoning_characters": 28451, "content_end": " \\times 1.020620726 \\approx 3,061,862.178 \\text{ km}\n\\]\n\nThe closest option to this value is D: 3,060,000 km.\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 181.3 s, median 173.8 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed9h 36m into the runtook 10m 19s
startup 142.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 142.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 1908, "reasoning_characters": 32189, "content_end": " = \\frac{45}{62.5} = \\frac{18}{25} = 0.72\n\\]\n\nThis ratio is approximately 0.7, which corresponds to option C.\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 182.3 s, median 174.2 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
I submitted the final model at 08:25 UTC, and the session is over. The model is Qwen3-4B-Base fine-tuned in three stages: long chain-of-thought training, extra public reasoning traces, then training to answer after the serving-time thinking budget closes the thought. The last stage also varies the multiple-choice answer format.
- Final selfcheck: Selfcheck #3 passed on this exact model. Startup took 142 s of the 300 s allowed. The slowest of 32 concurrent requests took 182 s of 250 s, with none timing out or stopping at max_tokens.
- Dev results: On my self-built dev set, the submitted model scored 0.593 on multiple-choice, 0.34 on math and 0.9875 on code. The earlier candidate scored 0.607, 0.373 and 0.9625. The differences are within sampling noise. The final model parsed 100% of multiple-choice answers under three answer-format variants, against 18–69% for the earlier one, which is why I chose it.
- What was graded:
/workspace/outputas it stood at submission, includingarchitecture.json, the provenance file and the logs. The selfcheck reported that the architecture review did not run, so that part is unchecked. I did not runvalidate.pyon the final model. Selection used only the self-built dev set, never the public benchmark splits.