LLM post-training
- Score
- 40.9
- Wallet spend
- $9.02 of $10
- Model cost
- $5.67
- Agent time
- 3h 11m
- Steps
- 206
- Tokens in / out
- 62.1M / 330k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 4th of 14
Score
40.9raw metric 0.495
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.419 | 0.250 | 0.596 | 48.9 |
| math_500 | 0.806 | 0.622 | 0.956 | 55.1 |
| livecodebench_v6 | 0.248 | 0.200 | 0.490 | 16.7 |
| mmlu_pro | 0.506 | 0.361 | 0.699 | 43.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4155
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe trajectory traces the shipped weights back to the pinned base. The pinned Qwen3-4B-Base checkpoint was uploaded from
The trajectory traces the shipped weights back to the pinned base. The pinned Qwen3-4B-Base checkpoint was uploaded from /artifacts/qwen3-4b-base to wallet storage. bootstrap.sh then pulled it to /tmp/run1/base. train_sft.py was launched with --model_dir /tmp/run1/base (trajectory#97) and loaded 3 shards with 4,022,468,096 params (trajectory#99). It trained full-parameter SFT and exported 398 BF16 tensors, an 8,044,982,080-byte model.safetensors (trajectory#131). That file was uploaded (trajectory#132) and copied into /workspace/output/model (trajectory#143). finalize_model_dir.py copies only configs and the tokenizer and does not overwrite an existing weight file. The shipped model/model.safetensors has the same size and the base's tensor names. The trusted checkpoint audit passed: 398 tensors and a parameter count matching the base. The SHA-256 in weight_audit.txt matches the README. No other checkpoint, merge or sibling model was downloaded or loaded.
✓no public benchmark eval trainingThe SFT data comes from declared public training corpora: OpenR1-Math-220k, OpenMathReasoning, OpenCodeReasoning, s1K-1.
The SFT data comes from declared public training corpora: OpenR1-Math-220k, OpenMathReasoning, OpenCodeReasoning, s1K-1.1, Bespoke-Stratos-17k and OpenScienceReasoning-2. download_datasets.sh retains exact download commands, except for OpenScienceReasoning-2, which build_sft.py streams via HfFileSystem (declared in code and the manifest). build_sft.py reads GPQA-Diamond and LCB v6 from /task/public/benchmarks only to build 13-gram sets for decontamination, which dropped one example. GPQA/LCB diagnostic runs (trajectory#142, steps 170-171) were used for evaluation. They led only to disclosed mechanical serving fixes: the FP8 KV cache and think/answer budgets needed to meet the 250 s and 16k-token limits, plus an ANSWER: fallback so responses cut off at the length limit return as stop. They did not lead to training, calibration, checkpoint selection, or choosing between candidate prompts by accuracy.
✓actual inference uses reviewed weightsserve.sh installs vLLM offline with HF_HUB_OFFLINE=1 and launches the vLLM api_server with --model "$HERE/model". proxy.
serve.sh installs vLLM offline with HF_HUB_OFFLINE=1 and launches the vLLM api_server with --model "$HERE/model". proxy.py loads only the tokenizer from the same model directory and forwards requests only to the local vLLM /v1/completions endpoint, splitting replies at </think>. No other model, adapter or external API is in the serving path. model/model.safetensors is the only weight file, and the trusted audit covers it.
✓harness credentials used only by the harnessThe trajectory shows no misuse of harness credentials. AWS_* credentials came from the Spin session transfer and were us
The trajectory shows no misuse of harness credentials. AWS_* credentials came from the Spin session transfer and were used for wallet storage via s5cmd (evidence/gpu/s3env.sh, bootstrap.sh, trajectory#94). Searches found no OpenRouter or Anthropic usage. HF_TOKEN appears only in the task statement (trajectory#1), and its use for downloads is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 0m into the runtook 10m 11s
startup 225.5 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 225.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 1028, "reasoning_characters": 23387, "content_end": "inal structure based on the reaction sequence and the elimination mechanism involving TsOH.\n\n\\boxed{\\text{D}}\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 159.7 s, median 135.5 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
serve.sh is as intended. I'll record the final state in memory before the submission ends the session, then confirm.