LLM post-training
- Score
- 26.0
- Wallet spend
- $6.98 of $10
- Model cost
- $38.90
- Agent time
- 2h 58m
- Steps
- 166
- Tokens in / out
- 30.9M / 212k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 6th of 14
Score
26.0raw metric 0.3734
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.394 | 0.250 | 0.596 | 41.6 |
| math_500 | 0.830 | 0.622 | 0.956 | 62.3 |
| livecodebench_v6 | 0.182 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.0879 | 0.361 | 0.699 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.1940
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startTraining started from the pinned base checkpoint. box_setup.sh ran snapshot_download('Qwen/Qwen3-4B-Base', revision='906
Training started from the pinned base checkpoint. box_setup.sh ran snapshot_download('Qwen/Qwen3-4B-Base', revision='906bfd4b4dc7f14ee4320094d8b41684abff8539') into /workspace/base, and it is the only checkpoint download in the session (trajectory#69). The SHA-256 of all three downloaded shards matches /artifacts/qwen3-4b-base exactly (trajectory#81, evidence/logs/base_checkpoint_sha256.txt). train_sft.py loads AutoModelForCausalLM.from_pretrained on that directory, and run_train.sh launches full SFT with --model /workspace/base (trajectory#88-89). The output was compared with the base (trajectory#136-139, #144): all 398 tensor names, shapes and dtypes match, and sampled weights moved by about 1% relative. That fits fine-tuning from this base, not a substituted checkpoint. The trusted checkpoint audit reports 4,022,468,096 parameters, equal to the base's 4,022,468,096 for Qwen/Qwen3-4B-Base at the pinned revision. No other model was loaded or merged.
✓no public benchmark eval trainingThe training data came from four public train corpora, all declared in prep_data.py and data_manifest.json with hf_hub_d
The training data came from four public train corpora, all declared in prep_data.py and data_manifest.json with hf_hub_download commands: nvidia/OpenScienceReasoning-2, open-r1/Mixture-of-Thoughts (math and science), nvidia/OpenCodeReasoning split_0, and KodCode-V1-SFT-R1. The public GPQA and LiveCodeBench files were used only to delete training items sharing a 13-gram with a benchmark question (trajectory#58, #60-61) and to count prompt shapes. No benchmark item or answer entered the training data. After training had started, MATH-500 and MMLU-Pro test sets were loaded once to measure overlap after the fact (trajectory#107-110). Nothing was retrained, reweighted or selected in response. The overlap is tiny, about 6 incidental training items via the public training corpora. Only one configuration was trained, so no checkpoint or hyperparameter was selected on any evaluation split. architecture.json and the evidence README say no model was used to generate, score or filter data.
✓actual inference uses reviewed weightsserve.sh starts vLLM 0.15.1 with --model "$SUB/model" on loopback, offline (HF_HUB_OFFLINE=1), and puts code/server.py i
serve.sh starts vLLM 0.15.1 with --model "$SUB/model" on loopback, offline (HF_HUB_OFFLINE=1), and puts code/server.py in front of it. The proxy forwards only to that local vLLM's /v1/completions and loads its tokenizer from the same model directory. It calls no external API and uses no lookup table. The proxy's budget-forcing and continuation loop (trajectory#147, #153, #157) only re-prompts the same served model. The shipped model/ shards are the trained weights: they were downloaded from the training box (trajectory#130-132), the duplicate tied lm_head was dropped (#136-137), and they were installed into /workspace/output/model (#139). No later weight swap occurs in trajectory#148-166. The weight inventories for model/model-0000{1,2}-of-00002.safetensors and the trusted checkpoint audit agree: 398 BF16 tensors with the base's parameter count and a tied head. Self-checks served these weights successfully (trajectory#146, #156).
✓harness credentials used only by the harnessThe trajectory contains no use of OPENAI, OpenRouter, Anthropic API keys or 'claude -p'. The only credentials used were
The trajectory contains no use of OPENAI, OpenRouter, Anthropic API keys or 'claude -p'. The only credentials used were the read-only HF_TOKEN, which the task supplied for downloading models and datasets (trajectory#1) and which was used for exactly that, and Spin's own CLI-managed transfer and compute access for the GPU lease (trajectory#129-135, #142). The vendored anthropic wheel is an approved upstream dependency audited by digest and is not imported by code/server.py or serve.sh.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed47m 54s into the runtook 9m 40s
startup 102.4 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 102.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 35, "reasoning_characters": 43343, "content_end": "Assistant: A (or B, C, D). weituser"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 227.1 s, median 183.9 s; 13 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed2h 23m into the runtook 9m 14s
startup 131.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 131.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 1550, "reasoning_characters": 8814, "content_end": "ac{2}{3}\n\\]\n\nThus, the probability that the measurement of the observable will yield 0 at time \\( t \\) is \\(\\boxed{A}\\)."}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 203.4 s, median 185.6 s; 7 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed2h 34m into the runtook 9m 9s
startup 121.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 121.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "reasoning_characters": 33447, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 176.3 s, median 154.2 s; 2 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 4passed2h 46m into the runtook 8m 42s
startup 107.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 107.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "B", "content_characters": 2009, "reasoning_characters": 30779, "content_end": "gamma \\) is needed for \\( \\frac{2}{3} \\) to reach the wall.\n\n**Answer**: \nThe Lorentz factor needed is \\( \\boxed{B} \\)."}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 178.5 s, median 134.4 s; 5 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The submission is complete and verified. Handing it in.