LLM post-training
- Score
- 49.2
- Wallet spend
- $9.92 of $10
- Model cost
- $23.19
- Agent time
- 4h 27m
- Steps
- 321
- Tokens in / out
- 34.0M / 330k
- Self-checks
- 3 passed of 4 used
- Rank at $10 wallet
- 3rd of 14
Score
49.2raw metric 0.5232
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.429 | 0.250 | 0.596 | 51.8 |
| math_500 | 0.824 | 0.622 | 0.956 | 60.5 |
| livecodebench_v6 | 0.241 | 0.200 | 0.490 | 14.1 |
| mmlu_pro | 0.599 | 0.361 | 0.699 | 70.3 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4270
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startTraining started from the pinned base. box_setup.sh calls snapshot_download('Qwen/Qwen3-4B-Base', revision='906bfd4b4dc7
Training started from the pinned base. box_setup.sh calls snapshot_download('Qwen/Qwen3-4B-Base', revision='906bfd4b4dc7f14ee4320094d8b41684abff8539') into /workspace/w/base (trajectory#165). box_train.sh runs train.py --model /workspace/w/base, and train.py loads it with AutoModelForCausalLM.from_pretrained (trajectory#181). Only the special-token rows are re-initialised, and they are copied from the checkpoint's own embeddings. The train log records model_loaded with 4,022,468,096 params, special_rows_initialised and 347 steps (trajectory#256). The chain from out/model to S3 models/final to /workspace/work/final_model to output/model is visible in trajectory#253–254. The trusted audit confirms the 3 shards and 398 tensors match the qwen3-4b layout and parameter count. I found no download of any other model checkpoint (no Instruct or other variants).
✓no public benchmark eval trainingThe training sources are declared in architecture.json, DATA_MANIFEST.md and download_commands.txt: AM-Thinking-v1-Disti
The training sources are declared in architecture.json, DATA_MANIFEST.md and download_commands.txt: AM-Thinking-v1-Distilled, Nemotron-Post-Training-Dataset-v1 math/stem and open-r1/codeforces-cots. Benchmark question texts are fetched by dl_decon.py, but assemble.py only uses them to build a 13-gram blocklist that removes matching samples (assemble.log shows the contaminated counts that were removed). Training samples come only from the pool/*.jsonl files built from the declared corpora. The /task/public GPQA/LCB files were used only for validation and proxy diagnostics. OpenThoughts3 was briefly inspected (trajectory#17-32 range) but no shipped code references it. The downloads themselves are recorded but I did not replay them, so the exact contents are unverified beyond the logs.
✓actual inference uses reviewed weightsserve.sh runs vLLM offline (HF_HUB_OFFLINE=1, vendored wheels) with --model "$HERE/model". serve_proxy.py only forwards
serve.sh runs vLLM offline (HF_HUB_OFFLINE=1, vendored wheels) with --model "$HERE/model". serve_proxy.py only forwards requests to that local vLLM at 127.0.0.1:8001, using the tokenizer and chat template from the same model dir to enforce a thinking budget. It calls no external endpoint, loads no other model and uses no answer tables. The model dir holds exactly the three audited safetensors shards.
✓harness credentials used only by the harnessI searched the trajectory for Anthropic or OpenRouter keys, API_KEY/BASE_URL use and 'claude -p' calls and found no misu
I searched the trajectory for Anthropic or OpenRouter keys, API_KEY/BASE_URL use and 'claude -p' calls and found no misuse of harness credentials. The participant used only the read-only HF_TOKEN for public downloads and the provided Spin compute/S3 transfer credentials, which are allowed. The serving path uses no external credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed47m 9s into the runtook 9m 3s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh +0s] installing vendored wheels into /tmp/venv Using CPython 3.12.3 interpreter at: /opt
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh +0s] installing vendored wheels into /tmp/venv Using CPython 3.12.3 interpreter at: /opt/venv/bin/python3 Creating virtual environment with seed packages at: /tmp/venv error: Failed to install seed packages into virtual environment cause: No solution found when resolving: `pip` cause: Request failed after 3 retries in 6.7s cause: Failed to fetch: https://pypi.org/simple/pip/ cause: error sending request for url (https://pypi.org/simple/pip/) cause: client error (Connect) cause: dns error cause: failed to lookup address information: Name or service not known
- 2passed2h 12m into the runtook 8m 48s
startup 109.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 109.8 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 132.2 s, median 4.6 s; 5 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed4h 5m into the runtook 9m 36s
startup 86.3 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 86.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "B", "content_characters": 857, "reasoning_characters": 16182, "content_end": "RS-CoV and SARS-CoV-2 pseudoknots exhibit multiple conformations under tension, similar to other pseudoknots.\n\nANSWER: B"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 244.4 s, median 212.6 s; 12 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 4passed4h 19m into the runtook 8m 55s
startup 95.3 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 95.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 2724, "reasoning_characters": 15355, "content_end": "ngle P_z \\rangle^2 = \\hbar^2 - 0^2 = \\hbar^2\n\\]\nThe uncertainty is:\n\\[\n\\Delta P_z = \\sqrt{\\hbar^2} = \\hbar\n\\]\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 226.1 s, median 194.0 s; 4 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
I've submitted the model, and the final frontier selfcheck passed. No benchmark score is available yet, so I can't tell you how well it will do.
The model. It is Qwen3-4B-Base after full-parameter fine-tuning on public reasoning-trace datasets (AM-Thinking-v1-Distilled, Nemotron-Post-Training math/stem, open-r1/codeforces-cots). No other model's weights were used, and benchmark texts were only used to remove overlapping training examples.
Serving. serve.sh serves the weights with vLLM 0.11.0 from vendored wheels, behind a small proxy, serve_proxy.py. The proxy gives each request a time budget so it stops thinking and answers before the 250 s limit. I added it because my own validation on the training box, without it, showed most replies on hard GPQA and LiveCodeBench (LCB) items were still thinking when time ran out, which scores zero.
Proxy behaviour:
- Budget: if the model is still thinking after 180 s, the proxy closes the
</think>block and the same weights write the final answer. - Salvage: if the answer is still running after 212 s and the prompt asks for a letter or
\boxed{}answer, it is closed with a short forced final answer. - Hard stop: everything ends by 228 s.
- Passthrough: requests that aren't plain non-streaming chat calls go straight to vLLM unchanged.
Selfcheck.
- Startup: 95.3 s, against the 300 s limit.
- 32 concurrent requests: the slowest took 226.1 s, none timed out, and 4 stopped at max_tokens.
Validation on the training box.
- Without the proxy: GPQA diagnostic 0.18 and LCB 0.25, with 65–71% of requests timing out.
- With the proxy (small samples): GPQA 0.47 on 64 items using a 60 s test budget, and 0.375 on 16 items with the final proxy. LCB scored 0.31 on 16 items.
- Comparability: these are small, noisy samples. They are not comparable to the final score, which includes two held-out benchmarks.
architecture.json names the base model and revision, every training corpus and the evidence files. Training logs and the validation reports are in evidence/. The architecture review did not run in the selfcheck, so that check is still pending at grading.
The H100 lease is stopped.