LLM post-training
- Score
- 55.2
- Wallet spend
- $49.80 of $50
- Model cost
- $18.87
- Agent time
- 3h 1m
- Steps
- 419
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 4 used
- Rank at $50 wallet
- 2nd of 19
Score
55.2raw metric 0.5429
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.444 | 0.250 | 0.596 | 56.2 |
| math_500 | 0.854 | 0.622 | 0.956 | 69.5 |
| livecodebench_v6 | 0.262 | 0.200 | 0.490 | 21.2 |
| mmlu_pro | 0.611 | 0.361 | 0.699 | 74.1 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4505
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startsrc/train_sft.py loads the starting weights with from_pretrained(args.base_model). The default is /artifacts/qwen3-4b-ba
src/train_sft.py loads the starting weights with from_pretrained(args.base_model). The default is /artifacts/qwen3-4b-base, the provided copy of Qwen/Qwen3-4B-Base at the pinned revision 906bfd4b4dc7f14ee4320094d8b41684abff8539, which architecture.json also declares. The trajectory shows that exact directory being uploaded to S3 and copied to /tmp/work/base_model. Every training run, from the smoke test to the final run, used --base-model /tmp/work/base_model. The weight soup averages only the run's own snapshots. In trajectory#359–#363 the final /tmp/work/output_model is uploaded and copied into /workspace/output/model, replacing an earlier intermediate copy. In trajectory#394 the md5 of the shipped shard matches the final output. The trusted checkpoint audit passed: 398 BF16 tensors and 4,022,468,096 parameters, matching the Qwen3-4B structure, with a small difference from the base (4.1e-3). No other checkpoint or instruct model was downloaded.
✓no public benchmark eval trainingsrc/download_data.py downloads only HF training splits: OpenThoughts-114k, OpenThoughts3-1.2M, OpenCodeReasoning split_0
src/download_data.py downloads only HF training splits: OpenThoughts-114k, OpenThoughts3-1.2M, OpenCodeReasoning split_0, the OpenMathReasoning cot split, and the Llama-Nemotron SFT science subset. All are declared in architecture.json and logs/data_manifest.json. src/prepare_data.py reads /task/public/benchmarks (gpqa_diamond.csv, livecodebench-v6) only to decontaminate, dropping 13,349 overlapping rows. The validation set (300 rows) is carved from the training data. No benchmark items, rewrites or answers enter training, validation or selection.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE=1 and MODEL_PATH=${SCRIPT_DIR}/model, then runs server.py. server.py builds a vLLM AsyncLLM
serve.sh sets HF_HUB_OFFLINE=1 and MODEL_PATH=${SCRIPT_DIR}/model, then runs server.py. server.py builds a vLLM AsyncLLMEngine with model and tokenizer set to MODEL_PATH, uses that single engine for both budget-forcing phases, and makes no external API calls and loads no other weights. The shipped model/ shards are the ones the trusted audit inventoried. All wheels are trusted-audit approved, and there are no uninspected files.
✓harness credentials used only by the harnessSearching the trajectory found no participant command that uses harness API keys or base URLs to call an external model.
Searching the trajectory found no participant command that uses harness API keys or base URLs to call an external model. Mentions of API_KEY/BASE_URL appear only in the Spin documentation text (trajectory#3). The 429 messages were harness system output (trajectory#358). HF_TOKEN appears only as a read header for downloading datasets in src/download_data.py.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 35m into the runtook 8m 5s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh] Installing vendored wheels from /submission/wheels into /tmp/site-packages... [serve.sh
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh] Installing vendored wheels from /submission/wheels into /tmp/site-packages... [serve.sh] Extracted 158 wheels into /tmp/site-packages in 16.40s [serve.sh] Starting inference server on 0.0.0.0:8000 for model submission... INFO 10-14 07:13:33 [__init__.py:216] Automatically detected platform cuda. [Server] Loading tokenizer and model from /submission/model... ERROR: Traceback (most recent call last): File "/tmp/site-packages/starlette/routing.py", line 654, in lifespan async with self.lifespan_context(app) as maybe_state: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 87, in lifespan engine_args = AsyncEngineArgs( ^^^^^^^^^^^^^^^^ TypeError: AsyncEngineArgs.__init__() got an unexpected keyword argument 'disable_log_requests' ERROR: Application startup failed. Exiting.
- 2failed2h 0m into the runtook 7m 38s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: t recent call last): File "/tmp/site-packages/starlette/routing.py", line 654, in lifespan a
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: t recent call last): File "/tmp/site-packages/starlette/routing.py", line 654, in lifespan async with self.lifespan_context(app) as maybe_state: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 99, in lifespan engine = AsyncLLMEngine.from_engine_args(engine_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/site-packages/vllm/engine/async_llm_engine.py", line 574, in from_engine_args return async_engine_cls.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/site-packages/vllm/utils/__init__.py", line 1589, in inner return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/tmp/site-packages/vllm/v1/engine/async_llm.py", line 212, in from_vllm_config return cls( ^^^^ File "/tmp/site-packages/vllm/v1/engine/async_llm.py", line 136, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/site-packages/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client return AsyncMPClient(*client_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/site-packages/vllm/v1/engine/core_client.py", line 769, in __init__ super().__init__( File "/tmp/site-packages/vllm/v1/engine/core_client.py", line 448, in __init__ with launch_core_engines(vllm_config, executor_class, File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/tmp/site-packages/vllm/v1/engine/utils.py", line 729, in launch_core_engines wait_for_engine_startup( File "/tmp/site-packages/vllm/v1/engine/utils.py", line 782, in wait_for_engine_startup raise R ...[clipped]
- 3passed2h 37m into the runtook 8m 24s
startup 100.2 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 22 well-formed replies; 10 timed out after 250 s. Grading scores each such request as wrong, so 10 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 100.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 2170, "reasoning_characters": 5139, "content_end": " is unrelated to the transcriptional or morphological processes directly triggered by the peptide.\n\n\\boxed{D}\n\nANSWER: D"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 251.2 s, median 247.8 s; 0 stopped at max_tokens, 10 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 22 well-formed replies; 10 timed out after 250 s. Grading scores each such request as wrong, so 10 of 32 here would score zero.
- ✓healthy after requests
- 4passed2h 51m into the runtook 5m 51s
startup 73.3 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 73.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 936, "reasoning_characters": 23260, "content_end": "focus, and other elements (e.g., promoters, origins) are irrelevant to the question. \n\n**Answer:** \\boxed{A}\n\nANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 94.9 s, median 85.0 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests