LLM post-training
- Score
- 16.1
- Wallet spend
- $7.42 of $50
- Model cost
- $18.88
- Agent time
- 7h 23m
- Steps
- 193
- Tokens in / out
- 25.8M / 63k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 11th of 14
Score
16.1raw metric 0.3504
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.348 | 0.250 | 0.596 | 28.5 |
| math_500 | 0.742 | 0.622 | 0.956 | 35.9 |
| livecodebench_v6 | 0.127 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.184 | 0.361 | 0.699 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.2284
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe trajectory traces the weights' full path. The participant uploaded the provided /artifacts/qwen3-4b-base (a 398-tens
The trajectory traces the weights' full path. The participant uploaded the provided /artifacts/qwen3-4b-base (a 398-tensor, 3-shard checkpoint) to the spin S3 prefix base/qwen3-4b-base (trajectory#15, trajectory#26/27 in steps-13-27). It was then pulled onto the GPU at /artifacts/qwen3-4b-base (trajectory#28). train_src/train_sft.py loads BASE=/artifacts/qwen3-4b-base with AutoModelForCausalLM.from_pretrained, runs full SFT, and saves to /workspace/output/model. The run log shows 'loading model from /artifacts/qwen3-4b-base' and 'saving to /workspace/output/model' (trajectory#47, #50, #128, #142). Only the base's own config and tokenizer files were copied next to the trained weights, and their hashes match the base (trajectory#117, #118). The trained export went to the S3 prefix 'trained' and was pulled into /workspace/output/model (trajectory#129, #136, #137, #138). Searches found no download of another model, no merge and no adapter. The trusted audit for checkpoint (qwen3-4b-inventory-v1) passed: 398 BF16 tensors, parameter count equal to the base Qwen/Qwen3-4B-Base@906bfd4b.
✓no public benchmark eval trainingtrain_src/prepare_data.py and DATA_MANIFEST.json declare only training splits of public corpora: OpenMathReasoning cot (
train_src/prepare_data.py and DATA_MANIFEST.json declare only training splits of public corpora: OpenMathReasoning cot (rows whose source contains test/valid/gpqa/mmlu are excluded), OpenR1-Math-220k, OpenCodeReasoning split_0, Magicoder-Evol-Instruct, MetaMathQA, SciQ train and OpenHermes-2.5. Each has a datasets.load_dataset command. The participant touched /task/public/benchmarks only once: reading 3 lines of gpqa_diamond.csv to check the prompt format (trajectory#179). The serving prompt and answer-extraction probes used synthetic questions (trajectory#119, #126, #127, #170 to #173). The selfcheck runs were harness smoke checks used to fix answer formatting, not training, calibration or checkpoint selection on benchmark data. There was only one training run, so no checkpoint was selected.
✓actual inference uses reviewed weightsserve.sh sets MODEL_PATH=${ROOT}/model with HF offline mode and runs serve.py. serve.py builds a vLLM AsyncLLMEngine wit
serve.sh sets MODEL_PATH=${ROOT}/model with HF offline mode and runs serve.py. serve.py builds a vLLM AsyncLLMEngine with model=MODEL_PATH and loads the tokenizer from the same path. Every completion, including the follow-up ANSWER pass, is generated by that local engine. The code makes no external API calls and uses no lookup tables. The model/model.safetensors weight inventory and the passed trusted checkpoint audit cover those weights.
✓harness credentials used only by the harnessSearches of the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL. The only credentials u
Searches of the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL. The only credentials used were spin transfer credentials (session-rw.json from 'spin transfer credentials'), used for S3 transfers, which the policy allows (trajectory#128, #137). HF_TOKEN was only checked for presence (trajectory#15). The served code contains no API clients.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed6h 40m into the runtook 8m 12s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: dotenv-1.2.4 python-json-logger-4.2.0 python-multipart-0.0.32 pyyaml-6.0.3 pyzmq-27.2.0 quack-kern
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: dotenv-1.2.4 python-json-logger-4.2.0 python-multipart-0.0.32 pyyaml-6.0.3 pyzmq-27.2.0 quack-kernels-0.6.5 referencing-0.37.0 regex-2026.9.29 requests-2.34.2 rich-15.0.0 rich-toolkit-0.20.5 rignore-0.8.1 rpds-py-2026.6.3 sentencepiece-0.2.2 sentry-sdk-2.71.0 setproctitle-1.3.8 shellingham-1.5.4 six-1.17.0 sniffio-1.3.1 sse-starlette-3.5.0 starlette-1.7.0 supervisor-4.3.0 tabulate-0.10.0 tiktoken-0.14.0 tokenizers-0.22.2 torch-c-dlpack-ext-0.1.5 tqdm-4.70.1 transformers-4.57.6 truststore-0.10.4 typer-0.27.2 typing-inspection-0.4.4 urllib3-2.8.0 uvicorn-0.54.0 uvloop-0.23.0 vllm-0.19.0 watchfiles-1.3.0 websockets-17.1 xgrammar-0.2.8 yarl-1.25.1 /submission/serve.py:246: DeprecationWarning: on_event is deprecated, use lifespan event handlers instead. Read more about it in the [FastAPI docs for Lifespan Events](https://fastapi.tiangolo.com/advanced/events/). @app.on_event("startup") INFO: Started server process [187] INFO: Waiting for application startup. loading /submission/model ERROR: Traceback (most recent call last): File "/opt/venv/lib/python3.12/site-packages/starlette/routing.py", line 654, in lifespan async with self.lifespan_context(app) as maybe_state: File "/opt/venv/lib/python3.12/site-packages/fastapi/routing.py", line 272, in __aenter__ await self._router._startup() File "/opt/venv/lib/python3.12/site-packages/fastapi/routing.py", line 6455, in _startup await handler() File "/submission/serve.py", line 249, in _startup await loop.run_in_executor(None, load_engine) File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/serve.py", line 230, in load_engine arg ...[clipped]
- 2passed6h 51m into the runtook 9m 11s
startup 196.0 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 196.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 18, "reasoning_characters": 1319, "content_end": "The answer is: 138"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 166.1 s, median 19.1 s; 10 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed7h 3m into the runtook 9m 22s
startup 202.4 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 202.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 21, "reasoning_characters": 294, "content_end": "The answer is: 100000"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 165.7 s, median 11.8 s; 10 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 4passed7h 12m into the runtook 10m 51s
startup 203.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 203.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 9, "reasoning_characters": 75785, "content_end": "ANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 177.3 s, median 28.5 s; 12 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The post-trained Qwen3-4B-Base checkpoint is submitted.
The weights are a full fine-tune of the pinned base checkpoint only, saved in bf16 with the original 398 tensor names and shapes. Training used public training splits and synthetic reasoning corpora (OpenMathReasoning, OpenR1-Math, OpenCodeReasoning split_0, Magicoder, MetaMathQA, SciQ train, OpenHermes), and no validation or test split of a public benchmark. serve.sh serves them through an OpenAI-compatible endpoint with vLLM.
The last grading-style check passed: the weight audit matched, /health came up in 204 seconds, and a science sample returned ANSWER: A. Twelve of 32 concurrent public requests still hit the 16,384-token cap, so those items score blank.