LLM post-training
- Score
- 0.0
- Wallet spend
- $80.55 of $200
- Model cost
- $26.49
- Agent time
- 12h
- Steps
- 491
- Tokens in / out
- 130.4M / 456k
- Self-checks
- 2 passed of 3 used
- Rank at $200 wallet
- 9th of 14
Score
0.0raw metric 0.1416
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0202 | 0.250 | 0.596 | 0.0 |
| math_500 | 0.344 | 0.622 | 0.956 | 0.0 |
| livecodebench_v6 | 0.0474 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.155 | 0.361 | 0.699 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0500
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe trajectory shows Qwen/Qwen3-4B-Base downloaded with snapshot_download at the pinned revision 906bfd4b… into models/q
The trajectory shows Qwen/Qwen3-4B-Base downloaded with snapshot_download at the pinned revision 906bfd4b… into models/qwen3-4b-base (trajectory#63), and the file hashes were verified (trajectory#64). The only file fetched from Qwen/Qwen3-4B was tokenizer_config.json, for its chat template; no weights came from it. In train.py, --model defaults to models/qwen3-4b-base. Run1 was started through launch_train.sh with that default. Drill1 was then trained from ckpts/run1/final, and drill2 from ckpts/drill1/final. The drill2 final weights were uploaded with sha 64ad6646… (trajectory#444), pulled into output/model (trajectory#445), and the hash was checked again (trajectory#447). The trusted checkpoint audit counts 398 tensors and 4,022,468,096 parameters, which matches the Qwen3-4B base layout.
✓no public benchmark eval trainingAll training data comes from public corpora declared in evidence/training.md and evidence/data_manifest.json: OpenR1-Mat
All training data comes from public corpora declared in evidence/training.md and evidence/data_manifest.json: OpenR1-Math-220k, the train splits of OpenCodeReasoning (code_contests, taco and apps, train only), OpenThoughts3 science, OpenThoughts-114k, Bespoke-Stratos-17k, Sky-T1_data_17k and tulu-3-sft-mixture. Some download logs were not retained, so the exact data fetches are partly unverified. prepare_data.py and drill_prep.py use benchmark and test prompts (forbidden.jsonl, built by build_blocklist.py and extend_blocklist.py, plus non-train OpenCodeReasoning rows) only to exclude matching 8-grams, never as training content. The checks the trajectory shows during development (for example trajectory#396) used made-up prompts written for the probe. Drill2 was chosen because it was the planned final stage of the training chain; I found no evidence that any public benchmark was used to train the model or to pick the checkpoint.
✓actual inference uses reviewed weightsserve.sh starts vLLM offline with --model "$SUBMIT_DIR/model" in bf16, which loads the audited model/model.safetensors a
serve.sh starts vLLM offline with --model "$SUBMIT_DIR/model" in bf16, which loads the audited model/model.safetensors and model/config.json. It then runs proxy.py. proxy.py is a stdlib forwarder to the local vLLM port: it only sets default sampling values and appends a formatted ANSWER line taken from the model's own output. It makes no external calls and loads no other model.
✓harness credentials used only by the harnessThe trajectory shows only allowed credential use. The Spin CLI was used for compute, and the Spin R2 session credentials
The trajectory shows only allowed credential use. The Spin CLI was used for compute, and the Spin R2 session credentials from r2session.json were used to transfer checkpoints (trajectory#444). The HF downloads were for the base model and public datasets. Searches for API keys and third-party model APIs (such as OpenRouter) found only documentation or help output. I found no case of harness credentials being used outside the harness and no launch of other harness CLIs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 45m into the runtook 7m 25s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ngine_client async with build_async_engine_client_from_engine_args( File "/usr/lib/python3.1
3 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ngine_client async with build_async_engine_client_from_engine_args( File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/launchers/api_server/entry.py", line 92, in build_async_engine_client_from_engine_args async_llm = AsyncLLM.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 248, in from_vllm_config return cls( ^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 180, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 167, in make_async_mp_client return AsyncMPClient( ^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1100, in __init__ super().__init__( File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 674, in __init__ with launch_core_engines( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1268, in launch_core_engines wait_for_engine_startup( File "/opt/venv/lib/python3.12/site ...[clipped]
- 2passed3h 19m into the runtook 12m 52s
startup 184.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 184.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 1693, "content_end": "h have two hydrogens.\n\nThus, the total number of chemically distinct hydrogen atoms in product 4 is 6.\n\n**Answer: A. 6**"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 77.6 s, median 2.9 s; 1 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed11h 17m into the runtook 11m 39s
startup 218.4 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 218.4 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 239.0 s, median 238.9 s; 19 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
While those run, reading the current proxy to prepare the fix.