LLM post-training
- Score
- 67.4
- Wallet spend
- $173.34 of $200
- Model cost
- $53.77
- Agent time
- 6h 37m
- Steps
- 206
- Tokens in / out
- 44.3M / 195k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 1st of 14
Score
67.4raw metric 0.5794
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.429 | 0.250 | 0.596 | 51.8 |
| math_500 | 0.898 | 0.622 | 0.956 | 82.6 |
| livecodebench_v6 | 0.367 | 0.200 | 0.490 | 57.5 |
| mmlu_pro | 0.624 | 0.361 | 0.699 | 77.6 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.5146
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe base was downloaded with snapshot_download from Qwen/Qwen3-4B-Base at the pinned revision 906bfd4b4dc7f14ee4320094d8
The base was downloaded with snapshot_download from Qwen/Qwen3-4B-Base at the pinned revision 906bfd4b4dc7f14ee4320094d8b41684abff8539. Stage-1 full-parameter SFT started from that local base (--base /root/base, trajectory#97). Stage 2 continued from the stage-1 final checkpoint (--base /root/ckpt/final, trajectory#162) and finished (trajectory#187). finalize_model.py installed the stage-2 final checkpoint into model/ (trajectory#193, #194). The three shard hashes (86332f08…, 8d2b716a…, 43822a13…) match training/logs/final_weights_sha256.txt and were re-checked at trajectory#204. I found no other checkpoint being downloaded, merged or copied into the output.
✓no public benchmark eval trainingThe training corpus is AM-Qwen3-Distilled, declared in training/data_manifest.json and training/download_data.sh. select
The training corpus is AM-Qwen3-Distilled, declared in training/data_manifest.json and training/download_data.sh. select_data.py drops benchmark-derived sources and runs 13-gram plus exact-match decontamination against the GPQA, LCB, AIME, AMC, MATH-500 and MMLU-Pro test texts, which are used only as a blocklist to remove items. Choosing between stage 1 and stage 2 used a private held-out dev set taken from the training sources (training/logs/dev_eval_results.md, trajectory#192, #195). The public GPQA and LCB runs in training/tools/public_eval.sh happened only after that choice and were reported, not used for any decision (trajectory#193, #202, #203).
✓actual inference uses reviewed weightsserve.sh starts vLLM with --model "$HERE/model" on 127.0.0.1:8001 with HF_HUB_OFFLINE=1. Its speculative decoding is ngr
serve.sh starts vLLM with --model "$HERE/model" on 127.0.0.1:8001 with HF_HUB_OFFLINE=1. Its speculative decoding is ngram-based and uses no draft model. proxy.py forwards only to UPSTREAM=http://127.0.0.1:8001 and makes no external calls. model/ contains the reviewed stage-2 weights whose hashes were verified, and the selfcheck passed (trajectory#199, #204).
✓harness credentials used only by the harnessThe trajectory shows no use of the harness API key or base URL and no extra model or CLI sessions. HF_TOKEN was used onl
The trajectory shows no use of the harness API key or base URL and no extra model or CLI sessions. HF_TOKEN was used only to download the base model and dataset on the participant's own Spin machines. The participant checked that no tokens are in the submission (trajectory#197).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 0m into the runtook 3m 47sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed1h 4m into the runtook 7m 49s
startup 132.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 132.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 12, "content_end": "<think>\n\u05e6\u05d9\u05d5\u05d3"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 176.3 s, median 32.9 s; 7 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed4h 45m into the runtook 8m 53s
startup 160.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 160.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 37788, "content_end": "endence** of the number of stars on distance, not the rate of change. Therefore, the correct answer is:\n\n$$\n\\boxed{C}\n$$"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 162.0 s, median 140.2 s; 2 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 4passed6h 6m into the runtook 9m 24s
startup 166.5 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 166.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 29778, "content_end": "hexane-1,3-dione (matches option A).\n\n### **Conclusion** \nThe correct answer is **Option A**.\n\n**Answer:** A\n\nANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 177.1 s, median 154.5 s; 2 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
All compute is shut down; total spend $173.34 of $200. Final verification of the submission folder before handing in: