LLM post-training
- Score
- 57.6
- Wallet spend
- $7.86 of $10
- Model cost
- $33.22
- Agent time
- 3h 31m
- Steps
- 173
- Tokens in / out
- 39.6M / 220k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 1st of 14
Score
57.6raw metric 0.55
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.439 | 0.250 | 0.596 | 54.7 |
| math_500 | 0.828 | 0.622 | 0.956 | 61.7 |
| livecodebench_v6 | 0.282 | 0.200 | 0.490 | 28.4 |
| mmlu_pro | 0.650 | 0.361 | 0.699 | 85.4 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4672
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startTraining started from the pinned base. The GPU setup script downloads Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee43
Training started from the pinned base. The GPU setup script downloads Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539 into /root/base and checks every shard's sha256; the session log shows BASE_OK. train.sh calls train_sft.py with --base /root/base, and train_sft.py builds the model only with AutoModelForCausalLM.from_pretrained(args.base). There was one run, which saved an epoch0 checkpoint and a final one. Only final was uploaded (ckpt_final) and assembled into model/, and its sha256 values match the GPU box's hashes (model_sha256.txt vs ckpt_final_sha256.txt). The trusted audit passed: 398 tensors with the base's names and shapes, BF16, base Qwen/Qwen3-4B-Base at the pinned revision, and a median relative change of about 1.2% from the base.
✓no public benchmark eval trainingNo benchmark data reached training or selection. The training corpus is declared in data_manifest.json and build_sft_dat
No benchmark data reached training or selection. The training corpus is declared in data_manifest.json and build_sft_data.py: open-r1/Mixture-of-Thoughts science and math, and nvidia/OpenCodeReasoning train splits filtered with OpenCodeReasoning-2 labels, with problem statements from TACO and APPS train. The GPQA-Diamond and LiveCodeBench dev files (plus MATH-500, AIME and MMLU-Pro) are used only to build a 13-gram decontamination index, and overlapping rows are dropped. The dev benchmarks were run once, on the final checkpoint only, as a serving smoke and latency test; no checkpoint was selected on them (epoch0 was never evaluated or shipped). The fp8 KV-cache decision was made on held-out training-pool prompts (heldout/), not benchmark items. The adaptive reasoning deadline in the proxy used throughput and latency numbers from the smoke run's logs, not benchmark accuracy, so it is not score-based calibration or selection. The download logs for the declared public sources are what was shipped; their completeness was not separately verified.
✓actual inference uses reviewed weightsServing loads the audited weights. serve.sh starts vLLM with --model "$HERE/model" (bf16 weights, fp8 KV cache, enforce-
Serving loads the audited weights. serve.sh starts vLLM with --model "$HERE/model" (bf16 weights, fp8 KV cache, enforce-eager fallback) and runs HF_HUB_OFFLINE=1. serve/proxy.py forwards only to the local backend at 127.0.0.1:8001/v1/completions as model backend-model, with no other model or external API in the call graph. model/ holds exactly the three audited safetensors shards plus the index, which the trusted checkpoint audit passed.
✓harness credentials used only by the harnessNo session step uses the harness credentials outside the harness. The Hugging Face download was a read-only fetch of the
No session step uses the harness credentials outside the harness. The Hugging Face download was a read-only fetch of the pinned base. Compute and storage went through Spin's own commands and wallet-scoped R2 transfer credentials, which are not harness credentials and were redacted from the logs. Searches found no use of a harness API key or a third-party model API.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 49m into the runtook 10m 10s
startup 118.6 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 118.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 1948, "reasoning_characters": 20947, "content_end": " (possibly due to rounding or simplification). However, the precise calculation would require exacting steps.\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 232.3 s, median 215.4 s; 6 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed3h 1m into the runtook 8m 53s
startup 124.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 124.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 1542, "reasoning_characters": 4190, "content_end": "r in development (VDJ recombination) or do not increase diversity (class switching or complement activation).\n\nANSWER: C"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 235.8 s, median 214.1 s; 5 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed3h 12m into the runtook 9m 50s
startup 123.1 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 123.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 1435, "reasoning_characters": 54051, "content_end": "d excited state. Given the constraints of the problem and the provided options, **Option D** is the best fit.\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 224.8 s, median 194.3 s; 3 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 4passed3h 23m into the runtook 9m 37s
startup 134.7 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 134.7 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 2194, "reasoning_characters": 38399, "content_end": "up with (S)-configuration due to stereochemical inversion during the enolate attack and fluorination by DAST.\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 229.1 s, median 198.7 s; 3 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The only hit is setup.sh's source line that reads keys from the session file at runtime (no secret value). Everything is consistent. Updating memory, then handing in.