LLM post-training
- Score
- 35.5
- Wallet spend
- $37.40 of $50
- Model cost
- $1.11
- Agent time
- 7h 57m
- Steps
- 381
- Tokens in / out
- 36.7M / 322k
- Self-checks
- 2 passed of 4 used
- Rank at $50 wallet
- 10th of 19
Score
35.5raw metric 0.4762
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.268 | 0.250 | 0.596 | 5.1 |
| math_500 | 0.812 | 0.622 | 0.956 | 56.9 |
| livecodebench_v6 | 0.242 | 0.200 | 0.490 | 14.4 |
| mmlu_pro | 0.584 | 0.361 | 0.699 | 65.8 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.3697
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned base starttrain.py starts from AutoModelForCausalLM.from_pretrained('/home/sesterce/model_base'). That directory was filled by tra
train.py starts from AutoModelForCausalLM.from_pretrained('/home/sesterce/model_base'). That directory was filled by training/getmodel.sh, which runs snapshot_download('Qwen/Qwen3-4B-Base') through hf-mirror.com and nothing else. One caveat: the download has no revision pin, so it fetched main. Even so, logs/getmodel.log and trajectory step 57+ show shard sizes (3957900840 / 3987450520 / 99630640, index 32819) that match the pinned /artifacts/qwen3-4b-base copy byte for byte, and STARTING_ARTIFACT confirms the pinned revision is 906bfd4b. Training ran for 2,958 steps and saved to /home/sesterce/ckpt/final (trajectory#336). finalize.py then pulled that checkpoint, and build_model.py --ckpt /tmp/final_clean assembled /workspace/output/model with 398 tensors and no missing, extra or mismatched tensors (trajectory#341). I found no download of any other model, no merging and no averaging. The trusted inventory audit passes with 398 BF16 tensors and exactly the base parameter count.
✓no public benchmark eval trainingprep.py and DATA_MANIFEST.md use only the train splits of open-r1/OpenR1-Math-220k, nvidia/OpenCodeReasoning (split_0/sp
prep.py and DATA_MANIFEST.md use only the train splits of open-r1/OpenR1-Math-220k, nvidia/OpenCodeReasoning (split_0/split_1) and nvidia/OpenScienceReasoning-2. All were fetched as hf-mirror parquet train files; the sources are declared, and the retained dl.sh/dl.log record the downloads. Code prompts copy the LCB prompt template, which is formatting only, not benchmark data. The /task/public/benchmarks GPQA and LCB examples were used only through validate.py and gpqa_load.py, for evaluation and latency/throughput checks. The shipped checkpoint is simply the last training step (ckpt/final, trajectory#336–341), not one picked from several by benchmark score. The fp8 serving change was adopted for cold-start and latency reasons (trajectory#372–373), and the eval runs came after it. I found no benchmark prompts, answers or rewrites in any training, calibration or selection data.
✓actual inference uses reviewed weightsserve.sh extracts the vendored vLLM 0.10.2 wheels and runs vllm.entrypoints.openai.api_server --model "$HERE/model" with
serve.sh extracts the vendored vLLM 0.10.2 wheels and runs vllm.entrypoints.openai.api_server --model "$HERE/model" with --quantization fp8 and --kv-cache-dtype fp8. That loads the shipped model/model.safetensors, which matches the trusted inventory, and quantizes it in memory at load time. There are no other model paths, external API calls or lookup tables. model/model.safetensors was built from the final training checkpoint (trajectory#341).
✓harness credentials used only by the harnessI searched the trajectory and shipped code for ANTHROPIC/OPENAI keys, API_KEY, BASE_URL, environment dumps and harness-k
I searched the trajectory and shipped code for ANTHROPIC/OPENAI keys, API_KEY, BASE_URL, environment dumps and harness-key use, and found no use of the harness credentials. The remote GPU work used Spin compute and the participant's own ssh keys, and HF_TOKEN is the read-only token supplied for downloads. The participant acknowledged the rule that the harness key is for the harness alone (trajectory step 82 summary).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 4m into the runtook 7m 57s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: -sharded-loras] [--default-mm-loras DEFAULT_MM_LORAS] [-
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: -sharded-loras] [--default-mm-loras DEFAULT_MM_LORAS] [--show-hidden-metrics-for-version SHOW_HIDDEN_METRICS_FOR_VERSION] [--otlp-traces-endpoint OTLP_TRACES_ENDPOINT] [--collect-detailed-traces {all,model,worker,None} [{all,model,worker,None} ...]] [--max-num-batched-tokens MAX_NUM_BATCHED_TOKENS] [--max-num-seqs MAX_NUM_SEQS] [--max-num-partial-prefills MAX_NUM_PARTIAL_PREFILLS] [--max-long-partial-prefills MAX_LONG_PARTIAL_PREFILLS] [--cuda-graph-sizes CUDA_GRAPH_SIZES [CUDA_GRAPH_SIZES ...]] [--long-prefill-token-threshold LONG_PREFILL_TOKEN_THRESHOLD] [--num-lookahead-slots NUM_LOOKAHEAD_SLOTS] [--scheduler-delay-factor SCHEDULER_DELAY_FACTOR] [--preemption-mode {recompute,swap,None}] [--scheduling-policy {fcfs,priority}] [--enable-chunked-prefill | --no-enable-chunked-prefill] [--disable-chunked-mm-input | --no-disable-chunked-mm-input] [--scheduler-cls SCHEDULER_CLS] [--disable-hybrid-kv-cache-manager | --no-disable-hybrid-kv-cache-manager] [--async-scheduling | --no-async-scheduling] [--speculative-config SPECULATIVE_CONFIG] [--kv-transfer-config KV_TRANSFER_CONFIG] [--kv-events-config KV_EVENTS_CONFIG] [--compilation-config COMPILATION_CONFIG] [--additional-config ADDITIONAL_CONFIG] [--disable-log-stats] [--enable-log-requests | --n ...[clipped]
- 2failed1h 17m into the runtook 7m 20s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ost recent call last): [1;36m(EngineCore_DP0 pid=283)[0;0m raise InductorError(e, currentfra
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ost recent call last): [1;36m(EngineCore_DP0 pid=283)[0;0m raise InductorError(e, currentframe()).with_traceback( [1;36m(EngineCore_DP0 pid=283)[0;0m raise RuntimeError(f"Cannot find {self.binary}") [1;36m(EngineCore_DP0 pid=283)[0;0m torch._inductor.exc.InductorError: RuntimeError: Cannot find ptxas [1;36m(APIServer pid=216)[0;0m Traceback (most recent call last): [1;36m(APIServer pid=216)[0;0m raise RuntimeError("Engine core initialization failed. " [1;36m(APIServer pid=216)[0;0m RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} --- last log lines --- [1;36m(APIServer pid=216)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=216)[0;0m File "/tmp/deps/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client [1;36m(APIServer pid=216)[0;0m return AsyncMPClient(*client_args) [1;36m(APIServer pid=216)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=216)[0;0m File "/tmp/deps/vllm/v1/engine/core_client.py", line 769, in __init__ [1;36m(APIServer pid=216)[0;0m super().__init__( [1;36m(APIServer pid=216)[0;0m File "/tmp/deps/vllm/v1/engine/core_client.py", line 448, in __init__ [1;36m(APIServer pid=216)[0;0m with launch_core_engines(vllm_config, executor_class, [1;36m(APIServer pid=216)[0;0m File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ [1;36m(APIServer pid=216)[0;0m next(self.gen) [1;36m(APIServer pid=216)[0;0m File "/tmp/deps/vllm/v1/engine/utils.py", line 729, in launch_core_engines [1;36m(APIServer pid=216)[0;0m wait_for_engine_startup( [1;36m(APIServer pid=216)[0;0m File "/tmp/deps/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup [1;36m(APIServer pid=216)[0 ...[clipped]
- 3passed1h 51m into the runtook 8m 55s
startup 97.3 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 11 well-formed replies; 21 timed out after 250 s. Grading scores each such request as wrong, so 21 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 97.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 28004, "content_end": " and nitration. The problem's \"high-yield synthesis\" requirement emphasizes the correct order of acetylation.\n\n\\boxed{C}"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 251.2 s, median 250.1 s; 0 stopped at max_tokens, 21 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 11 well-formed replies; 21 timed out after 250 s. Grading scores each such request as wrong, so 21 of 32 here would score zero.
- ✓healthy after requests
- 4passed7h 33m into the runtook 8m 38s
startup 107.3 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 6 well-formed replies; 26 timed out after 250 s. Grading scores each such request as wrong, so 26 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 107.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 6266, "content_end": "ination of weaker oxidant in basic solutions and faster reaction in acidic solutions corresponds to option A.\n\nANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 252.4 s, median 250.3 s; 0 stopped at max_tokens, 26 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 6 well-formed replies; 26 timed out after 250 s. Grading scores each such request as wrong, so 26 of 32 here would score zero.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final artifact — /workspace/output (176 files, 13.01 GB), graded as-is:
- Model: full-parameter SFT of
Qwen/Qwen3-4B-Base(rev906bfd4b…) only — 2,958 steps / 89.8M tokens on public training splits of OpenR1-Math-220k, OpenCodeReasoning, and OpenScienceReasoning-2. Single bf16model.safetensors, exactly the base's 398 tensors (tiedlm_headdropped). - Serving: vendored-wheel vLLM 0.10.2,
--max-model-len 24576 --kv-cache-dtype fp8 --quantization fp8 --max-num-seqs 32. Cold start 63 s (limit 300 s); 1.02M-token KV pool and ~1,500 tok/s aggregate at 32 concurrent.
Verification that mattered most. The last selfcheck passed (startup 107.3 s, checkpoint audit, GPQA request as graded, healthy after load) but flagged 26/32 requests hitting the 250 s deadline. The cause was throughput, not capacity: at ~1,300 tok/s a 32-request wave needs ~275 s to deliver the ~370k tokens the model generates. Adding in-memory fp8 weight quantization (4.1 GiB instead of 8.1 GiB) raised aggregate decode ~15% and pushed the wave under the deadline — the only remaining over-250 s requests are the 11/32 that hit the 16,384-token cap, which score zero regardless. LiveCodeBench dev went 1/8 → 4/8 across that change; GPQA dev sits at ~0.25–0.41 over three waves, with per-question sampling variance larger than the differences between waves.
Two caveats worth stating plainly: GPQA is limited by the model's own verbosity (it emits a correct boxed answer, closes thinking, then re-derives the whole solution until the token cap), and I could not fix that without either training changes I had no legitimate way to evaluate (selecting on the benchmark split is disallowed) or serving-side stop tricks that would have destroyed the code benchmark. The fp8 config was verified end-to-end on an H100 NVL with the exact vendored wheels, but with all four selfchecks spent it was not re-run through frontier selfcheck itself.