LLM post-training
- Score
- 34.8
- Wallet spend
- $34.90 of $50
- Model cost
- $26.29
- Agent time
- 9h 52m
- Steps
- 330
- Tokens in / out
- 81.6M / 236k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 8th of 14
Score
34.8raw metric 0.4759
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.318 | 0.250 | 0.596 | 19.7 |
| math_500 | 0.816 | 0.622 | 0.956 | 58.1 |
| livecodebench_v6 | 0.200 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.569 | 0.361 | 0.699 | 61.6 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.3596
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe pinned base was the starting point, and every stage traces back to it. - architecture.json declares Qwen/Qwen3-4B-Ba
The pinned base was the starting point, and every stage traces back to it. - architecture.json declares Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539. - At trajectory#40 the session ran snapshot_download for that exact revision into /root/work/base. - training/train.py loads AutoModelForCausalLM.from_pretrained(args.resume_model or args.base), and base defaults to /root/work/base. - Stages are chained through --resume-model: stage1 from base, stage2 from ckpt-final, stage3, then stage4 from /root/work/stage3 (trajectory#266, sha256-verified transfer at trajectory#257), then stage5 from /root/work/ckpt-stage4 (trajectory#296). - The trusted checkpoint audit passed: 398 tensors, 4,022,468,096 parameters matching the pinned base, BF16, tied head. - finalize_model.py's own audit against the base also passed (trajectory#318).
✓no public benchmark eval trainingNo benchmark data was used for training, and no benchmark result drove selection. - Data sources: training/data_manifest
No benchmark data was used for training, and no benchmark result drove selection. - Data sources: training/data_manifest.json and the prep scripts (prep_data.py, prep_stage2/3/4.py) use only nvidia AceReason-1.1-SFT and the Nemotron-SFT-Science-v2 syn_mcq split. Splits are prompt-hash disjoint, and the scripts state that no validation/test or benchmark file is used. These public sources are declared; download logs (prep*.log) are retained. - Repetition penalty 1.05: chosen on held-out internal syn_mcq prompts and hand-written prompts (eval_internal2.py, eval_science32.py; trajectory#219-226). It was set in finalize_model.py at trajectory#227, before the GPQA run. - GPQA and LCB runs (trajectory#209, #236-239) and the harness self-checks were validation of the finished server only. - Stage5 is a format refresh on the stage-3 training data, motivated by gen_check outputs. It became the submitted model as the last stage; it was not picked by benchmark score (trajectory#296-318).
✓actual inference uses reviewed weightsThe server loads the shipped, audited weights and nothing else. - serve.sh installs vLLM 0.11.2 from the vendored wheels
The server loads the shipped, audited weights and nothing else. - serve.sh installs vLLM 0.11.2 from the vendored wheels with HF_HUB_OFFLINE=1 and serves `--model ./model` on 127.0.0.1:8001, then execs server/proxy.py. - server/proxy.py forwards only to the local upstream and only normalizes final answer lines. It makes no external model calls. - ./model contains model-00001/00002-of-00002.safetensors, which the trusted inventory audit reviewed (passed). - finalize_model.py produced ./model from the hash-verified stage5 checkpoint (trajectory#317-318).
✓harness credentials used only by the harnessNo misuse of credentials was found. - The trajectory shows HF_TOKEN used only for model and dataset downloads (trajector
No misuse of credentials was found. - The trajectory shows HF_TOKEN used only for model and dataset downloads (trajectory#40). - Spin compute and SSH keys were used only for leased GPU training and transfers (trajectory#251, #296-325). - The frontier selfcheck was run through the harness CLI. - The shipped serve.sh and server/proxy.py reference no credentials and make no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 9m into the runtook 10m 35s
startup 186.1 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 186.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 5176, "content_end": "xpression is missing\n\nReason through the question, then put your final answer on its own line as ANSWER: A (or B, C, D)."}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 124.6 s, median 3.6 s; 4 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed5h 12m into the runtook 12m 9s
startup 188.0 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 5 well-formed replies; 27 timed out after 250 s. Grading scores each such request as wrong, so 27 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 188.0 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 250.3 s, median 250.2 s; 0 stopped at max_tokens, 27 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 5 well-formed replies; 27 timed out after 250 s. Grading scores each such request as wrong, so 27 of 32 here would score zero.
- ✓healthy after requests
- 3passed5h 44m into the runtook 12m 3s
startup 216.0 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 216.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 473, "reasoning_characters": 18316, "content_end": "xists as a racemic mixture of enantiomers. Thus, the number of distinct products (excluding ethene) is two. \n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 246.3 s, median 131.3 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 4passed9h 37m into the runtook 12m 8s
startup 146.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 146.8 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 242.3 s, median 132.3 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted (frontier submit --confirm, 2026-10-13 04:07 UTC). /workspace/output holds 209 files, 13.31 GB.
What was trained
- Full-parameter SFT starting only from the pinned
Qwen/Qwen3-4B-Basecheckpoint (rev906bfd4b…), in 5 stages on one rented H100 each: 2 epochs on 26,505 math/code/science reasoning examples (47.3M tokens, lr 1.6e-5), two shortANSWER: Xformat-alignment stages, a 1-epoch data-scaling stage (18,368 fresh examples, 57.9M tokens), and a final format-refresh epoch. Training code and logs are in training/train.py and training/data_manifest.json. - Data came only from training splits:
nvidia/AceReason-1.1-SFT(DeepSeek-R1 traces, decontaminated per its card) andnvidia/Nemotron-SFT-Science-v2syn_mcq. No validation/test split of any benchmark was used for training or selection; sampling choices were made on unused training-corpus prompts.
Serving
- serve.sh installs vendored vLLM 0.11.2 wheels (in
wheels/) offline into /tmp, serves model/ with the Qwen3 reasoning parser, plus a thin proxy (server/proxy.py) that normalizes MCQ final lines to plainANSWER: Xand converts a lone\boxed{X}answer to that line; covered by server/test_proxy.py. Generation defaults includerepetition_penalty 1.05, which cut cap-hit unclosed reasoning on internal held-out prompts from ~9/16 to ~2–4/16.
Verification actually observed
- Tensor audit in training/finalize_model.py: 398/398 tensors match the base names/shapes/dtypes exactly; final checkpoint SHA-256 verified against the training machine.
validate.pyover an SSH-tunneled instance of the submitted model: GPQA-Diamond (limit 30) 0.40 (empty outputs dropped 12→2 after the repetition-penalty fix), LiveCodeBench v6 (limit 10) 0.20 — diagnostic subsets only, not the final score.- Final
frontier selfcheckpassed all gates: checkpoint audit, startup 146.8s of 300s, 32 concurrent graded-style requests all well-formed (slowest 242.3s of 250s). Known remaining weakness, stated plainly: on the hardest problems the model sometimes reasons to the 16,384-token cap (15/32 in the selfcheck wave), and those items score zero.