LLM post-training
- Score
- 51.6
- Wallet spend
- $9.31 of $10
- Model cost
- $51.95
- Agent time
- 2h 44m
- Steps
- 250
- Tokens in / out
- 37.8M / 123k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
51.6raw metric 0.5314
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.439 | 0.250 | 0.596 | 54.7 |
| math_500 | 0.822 | 0.622 | 0.956 | 59.9 |
| livecodebench_v6 | 0.241 | 0.200 | 0.490 | 14.1 |
| mmlu_pro | 0.624 | 0.361 | 0.699 | 77.6 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4325
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base starttraining/train.py loads only the base, using AutoModelForCausalLM.from_pretrained(a.base, local_files_only=True). That b
training/train.py loads only the base, using AutoModelForCausalLM.from_pretrained(a.base, local_files_only=True). That base is the supplied pinned artifact /artifacts/qwen3-4b-base. It was uploaded to s3://spin/base at trajectory#15 and #19. The remote init log at trajectory#217 records base=/tmp/qwenwork/base, the pinned revision 906bfd4b and the base shard hashes. Training finished at step 897 (trajectory#212). The exported model went to s3://spin/result_model and was downloaded to /workspace/final_model (trajectory#213-215, #220). At trajectory#221 it was installed into output/model only after all three shard sha256 hashes matched the remote training export. That step replaced an interim recovery600 checkpoint, which was itself the candidate's own run. The earlier interim copies at trajectory#95 and #176 were also the candidate's own recovery checkpoints. The audit at trajectory#221/#226 and the trusted checkpoint audit show 398 tensors with Qwen3-4B names and shapes, 4.02B parameters, and 344 tensors changed from the base. I found no merge, substitution or post-trained initialization.
✓no public benchmark eval trainingtraining/download_data.py fetches open-r1 codeforces-cots, OpenR1-Math-220k and Mixture-of-Thoughts at pinned dataset re
training/download_data.py fetches open-r1 codeforces-cots, OpenR1-Math-220k and Mixture-of-Thoughts at pinned dataset revisions. These sources are declared in architecture.json. training/prepare_data.py reads only train-*.parquet files. It also applies a BLOCK regex that drops GPQA, LiveCodeBench, MATH-500, AIME and AMC items. The public validator (validate.py, --limit 16) ran only after the final step-897 checkpoint was hash-verified and installed (trajectory#221-#235). The candidate states that public evaluation did not change the recipe, checkpoint or decoding settings. The final step was fixed by schedule, and I found no evidence that any checkpoint or prompt was chosen by public benchmark scores. I could not verify the dataset downloads themselves beyond the shipped scripts and logs.
✓actual inference uses reviewed weightsserve.sh installs local wheels and runs exec python serve.py --model $ROOT_DIR/model --served-model-name submission with
serve.sh installs local wheels and runs exec python serve.py --model $ROOT_DIR/model --served-model-name submission with the deepseek_r1 reasoning parser. serve.py only patches the reasoning parser's fallback and then starts vLLM's OpenAI api_server. It loads no other model and makes no remote calls. The served weights are the model/model-0000{1,2,3}-of-00003.safetensors shards that the trusted audit checked: 398 BF16 tensors, passed.
✓harness credentials used only by the harnessOPENAI_API_KEY appears only in an environment-variable name listing at trajectory#7. Searching the evidence turned up no
OPENAI_API_KEY appears only in an environment-variable name listing at trajectory#7. Searching the evidence turned up no other use of it. The session used the Spin/frontier CLIs and a spin transfer session for S3 (training/transfer.py, trajectory#24, #26), both of which are allowed. architecture.json declares no teacher or API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed19m 26s into the runtook 11m 54s
startup 88.4 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 28 timed out after 250 s; 4 failed after up to 3 attempts: malformed response: choice has no text content | malformed response: choice has no text content | malformed response: choice has
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 88.4 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "reasoning_characters": 42197, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 250.4 s, median 250.3 s; 0 stopped at max_tokens, 28 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 28 timed out after 250 s; 4 failed after up to 3 attempts: malformed response: choice has no text content | malformed response: choice has no text content | malformed response: choice has no text content || malformed response: choice has no text content | malformed response: choice has no text content | malformed response: choice has no text content || malformed response: choice has no text content | malformed response: choice has no text content | malformed response: choice has no text content. Grading scores each such request as wrong, so 32 of 32 here would score zero.
- ✓healthy after requests
- 2passed2h 36m into the runtook 9m 46s
startup 96.9 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 15 well-formed replies; 17 timed out after 250 s. Grading scores each such request as wrong, so 17 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 96.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 1681, "reasoning_characters": 28609, "content_end": "onfiguration results in the correct number of loops.\n\nThus, the number of loops in the Feynman diagram is \\(\\boxed{C}\\)."}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 253.2 s, median 250.4 s; 0 stopped at max_tokens, 17 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 15 well-formed replies; 17 timed out after 250 s. Grading scores each such request as wrong, so 17 of 32 here would score zero.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with trained weights, offline serve.sh, and training evidence.
- Trained 897 steps on 44.1 million tokens.
- Weight audit passed.
- Final self-check passed; startup took 96.9 seconds.
- Limitation: 17/32 concurrent requests exceeded the 250-second deadline.