LLM post-training
- Score
- 58.1
- Wallet spend
- $111.68 of $200
- Model cost
- $33.59
- Agent time
- 3h 57m
- Steps
- 475
- Tokens in / out
- — / —
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 4th of 19
Score
58.1raw metric 0.553
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.490 | 0.250 | 0.596 | 69.3 |
| math_500 | 0.834 | 0.622 | 0.956 | 63.5 |
| livecodebench_v6 | 0.259 | 0.200 | 0.490 | 20.3 |
| mmlu_pro | 0.629 | 0.361 | 0.699 | 79.3 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4601
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe provided base at /artifacts/qwen3-4b-base was uploaded to s3://spin/qwen3-4b-base (trajectory#60, trajectory#63). It
The provided base at /artifacts/qwen3-4b-base was uploaded to s3://spin/qwen3-4b-base (trajectory#60, trajectory#63). It was used as /root/qwen3-4b-base. Stage 1 in src/run_train_pipeline.sh starts from that base (--model-path /root/qwen3-4b-base --init-special-tokens). Stage 2 starts from the stage 1 EMA. src/merge_and_select.py soups and selects only the participant's own stage checkpoints. src/run_stage3.sh then fine-tunes final_model and soups the result into final_model_v2, which was copied into model/ (trajectory#455). In src/train_sft.py, the special-token init reuses the base's own embeddings, and the save checks every key and shape against the base. I found no download of any other post-trained checkpoint. The trusted checkpoint audit passed: 398 BF16 tensors, about 4.02B parameters, in Qwen3-4B shapes.
✓no public benchmark eval trainingsrc/download_datasets.py, src/prepare_data.py and src/prepare_stage3.py pull only from the corpora declared in architect
src/download_datasets.py, src/prepare_data.py and src/prepare_stage3.py pull only from the corpora declared in architecture.json: Mixture-of-Thoughts train shards, nvidia/OpenScience, Nemotron-Science-v1 MCQ/RQA, OpenCodeReasoning split_0 kept only where split=='train', OpenThoughts-114k-Code_decontaminated, and OpenThoughts-114k shard 11 (camelai/riddle_sense rows). No source file loads GPQA, LiveCodeBench, AIME or MATH-500 test data. Checkpoint selection uses sft_val.pt, which is carved out of these training splits. The download commands for the two OpenThoughts sets are not in the shipped downloader, so their downloads are unverified, but both are declared. The supplied dev benchmarks were used only for validation and format tests. The only OpenRouter mention in the trajectory is CLI help text.
✓actual inference uses reviewed weightsserve.sh starts a local vLLM OpenAI server with --model ${SCRIPT_DIR}/model on port 18001, then runs src/proxy_server.py
serve.sh starts a local vLLM OpenAI server with --model ${SCRIPT_DIR}/model on port 18001, then runs src/proxy_server.py pointed at http://127.0.0.1:18001. The proxy loads only the tokenizer from model/ and forwards chat and completion requests (including budget-forcing continuations) only to that local backend. It makes no external calls and contains no hardcoded answers. model/ holds the audited 3-shard safetensors checkpoint.
✓harness credentials used only by the harnessIn the trajectory, the participant's commands use only the spin CLI and s5cmd with the participant's own wallet session
In the trajectory, the participant's commands use only the spin CLI and s5cmd with the participant's own wallet session (trajectory#60), plus HF downloads. The Gemini/antigravity-cli strings are the harness's own task-log paths. No step reads or uses the harness's model API keys or base URLs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 34m into the runtook 8m 16s
startup 48.4 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 6 well-formed replies; 26 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | no time left in the self-check for a retry || RuntimeError: Submission returned HTT
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 48.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "B", "content_characters": 709, "reasoning_characters": 2420, "content_end": "prox 2.5m \\cdot c^2 + 3.464m \\cdot c^2 \\approx 5.964m \\cdot c^2.\n\\] \nThe correct answer is **B**.\n\n\\boxed{B}\n\nANSWER: B"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 245.3 s, median 245.3 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 6 well-formed replies; 26 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | no time left in the self-check for a retry. Grading scores each such request as wrong, so 26 of 32 here would score zero.
- ✓healthy after requests
- 2passed2h 56m into the runtook 5m 45s
startup 45.3 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 45.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 1271, "reasoning_characters": 10794, "content_end": "ormat incompatibility is mitigated by standardized tools. \n\n**Answer:**\n\nTherefore, the answer is \\boxed{C}.\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 113.7 s, median 96.5 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed3h 51m into the runtook 5m 13s
startup 45.5 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 45.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 1826, "reasoning_characters": 11629, "content_end": "article masses, but it remains continuous.\n\nThus, the correct answer is:\n\nTherefore, the answer is \\boxed{C}.\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 84.5 s, median 57.9 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests