LLM post-training
- Score
- 26.1
- Wallet spend
- $41.88 of $50
- Model cost
- $24.50
- Agent time
- 10h 19m
- Steps
- 379
- Tokens in / out
- 69.0M / 165k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 9th of 14
Score
26.1raw metric 0.4435
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.258 | 0.250 | 0.596 | 2.2 |
| math_500 | 0.800 | 0.622 | 0.956 | 53.3 |
| livecodebench_v6 | 0.191 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.526 | 0.361 | 0.699 | 48.7 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.3256
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json declares Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539, and the trajectory s
architecture.json declares Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539, and the trajectory shows that exact base being downloaded to /home/sesterce/base. Run A started from it (trajectory#126: train_sft.py --base /home/sesterce/base). Run B started from Run A's output (trajectory#251-329: --base /home/sesterce/work/sftA/final). evidence/train_sft.py loads the starting weights with AutoModelForCausalLM.from_pretrained(args.base), and evidence/package_model.py packages sftB/final as pkgB. No other checkpoint download (Instruct or anything else) appears. The trusted audit of the checkpoint matches the shipped weights to this base: 398 BF16 tensors, a tied head, and 4,022,468,096 parameters, identical to the base's count. trajectory#331-333 also shows the tensor names and shapes match the base.
✓no public benchmark eval trainingevidence/DATA_MANIFEST.md declares the training data as public NVIDIA sets: OpenMathReasoning, OpenCodeReasoning and Ope
evidence/DATA_MANIFEST.md declares the training data as public NVIDIA sets: OpenMathReasoning, OpenCodeReasoning and OpenScienceReasoning-2. The prep scripts and bench_hashes.py use GPQA and LCB only for 8-gram decontamination, through contaminated(). Checkpoint selection (Run A vs Run B) and decoding selection (rep 1.05 / top_p 0.9, rejecting fp8 KV, the 10240-token cap) were made on dev_clean.jsonl, a held-out slice of the training corpus (trajectory#305-322, #368-369, evidence/SERVING_NOTES.md), plus the harness selfcheck, which checks timeouts. The one public-benchmark run, validate.py on 40 GPQA items (trajectory#341-344), came after pkgB was finalized. It was used only to diagnose problems and did not change any checkpoint or prompt choice. No benchmark data was used for training. Download logs (dl2.log) are included in the shipped files.
✓actual inference uses reviewed weightsserve.sh runs vLLM's OpenAI server offline with --model "$SUB/model". It loads no other model, makes no external API cal
serve.sh runs vLLM's OpenAI server offline with --model "$SUB/model". It loads no other model, makes no external API calls, and the shipped version uses bf16 KV (the fp8 flag was reverted in trajectory#368). The weights in output/model came from pkgB, copied via s3 (trajectory#329). The trusted checkpoint audit and the weight inventories of model/model-00001-of-00002.safetensors and model-00002-of-00002.safetensors cover exactly those files.
✓harness credentials used only by the harnessThe trajectory contains no step where the participant used harness API keys, base URLs or OpenRouter credentials. Search
The trajectory contains no step where the participant used harness API keys, base URLs or OpenRouter credentials. Searches for API_KEY, BASE_URL, openrouter and environ matched only Spin documentation and metadata. The only credentials seen are Spin transfer credentials, used for the participant's own s3 transfers (trajectory#325), and the normal HF downloads. Both are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed9h 10m into the runtook 11m 28s
startup 114.0 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 14 well-formed replies; 18 timed out after 250 s. Grading scores each such request as wrong, so 18 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 114.0 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 251.9 s, median 250.8 s; 0 stopped at max_tokens, 18 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 14 well-formed replies; 18 timed out after 250 s. Grading scores each such request as wrong, so 18 of 32 here would score zero.
- ✓healthy after requests
- 2passed9h 54m into the runtook 10m 39s
startup 107.0 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 8 well-formed replies; 24 timed out after 250 s. Grading scores each such request as wrong, so 24 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 107.0 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 253.1 s, median 250.4 s; 0 stopped at max_tokens, 24 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 8 well-formed replies; 24 timed out after 250 s. Grading scores each such request as wrong, so 24 of 32 here would score zero.
- ✓healthy after requests
- 3passed10h 7m into the runtook 9m 9s
startup 109.3 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 109.3 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 148.8 s, median 147.9 s; 22 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Everything is verified and in place. Submitting: