LLM post-training
- Score
- 12.9
- Wallet spend
- $197.37 of $200
- Model cost
- $47.57
- Agent time
- 11h 26m
- Steps
- 491
- Tokens in / out
- 109.4M / 228k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 8th of 14
Score
12.9raw metric 0.3873
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.253 | 0.250 | 0.596 | 0.7 |
| math_500 | 0.766 | 0.622 | 0.956 | 43.1 |
| livecodebench_v6 | 0.144 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.386 | 0.361 | 0.699 | 7.6 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.2704
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json declares Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539. The trajectory shows
architecture.json declares Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539. The trajectory shows that revision is the only base checkpoint that was downloaded. The full-parameter SFT chain is: phase 1 trained from the base directory (train_sft.py packed_full base ckpt), phase 2 from /workspace/ckpt/final, and phase 3 from /workspace/model. /workspace/model is the phase-2 package pulled from s3 models/model_dir. The phase-3 output was packaged, audited against the pinned base, uploaded and copied into output/model. The trusted checkpoint audit passed with 398 BF16 tensors and a parameter count equal to the pinned base. No other Qwen3 variant checkpoint was used as an initialization.
✓no public benchmark eval trainingevidence/build_data.py builds SFT data only from declared public training corpora: OpenMathReasoning, OpenCodeReasoning
evidence/build_data.py builds SFT data only from declared public training corpora: OpenMathReasoning, OpenCodeReasoning (train split), SCP-116K and OpenThoughts3. It also uses teacher MCQ traces from the train splits of sciq, ai2_arc, openbookqa, medmcqa and the mmlu auxiliary_train set (evidence/gen_mcq.py). The GPQA and LiveCodeBench dev files in /task/public/benchmarks are read only to build n-gram blocklists that remove contaminated rows. The trajectory shows them used only for probing and evaluation, never as training rows. Sources are declared in evidence/DATA_MANIFEST.md. Their exact download logs were not checked line by line and are recorded here as unverified rather than violating.
✓actual inference uses reviewed weightsserve.sh launches the vLLM OpenAI server (vendored wheels) with --model "$SUB/model". That is the audited two-shard safe
serve.sh launches the vLLM OpenAI server (vendored wheels) with --model "$SUB/model". That is the audited two-shard safetensors checkpoint whose inventory passed the trusted audit. code/sitecustomize.py only registers a <think>/</think> reasoning parser and makes no model or network calls. No external API, adapter or alternative model sits in the serving path, and there are no uninspected files.
✓harness credentials used only by the harnessThe teacher-data generation in gen_mcq.py uses a key created by the participant with `spin openrouter key create` (traje
The teacher-data generation in gen_mcq.py uses a key created by the participant with `spin openrouter key create` (trajectory#42). It calls the spin proxy base URL, not the harness's own credentials. No trajectory step was found that reads or uses the harness API_KEY/BASE_URL for the participant's own calls, and serving makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed7h 8m into the runtook 14m 39s
startup 146.0 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 11 well-formed replies; 19 timed out after 250 s; 2 failed after up to 3 attempts: malformed response: choice has no text content | malformed response: choice has no text content | malformed response: choice ha
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 146.0 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 251.1 s, median 250.0 s; 2 stopped at max_tokens, 19 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 11 well-formed replies; 19 timed out after 250 s; 2 failed after up to 3 attempts: malformed response: choice has no text content | malformed response: choice has no text content | malformed response: choice has no text content || malformed response: choice has no text content | malformed response: choice has no text content | malformed response: choice has no text content. Grading scores each such request as wrong, so 21 of 32 here would score zero.
- ✓healthy after requests
- 2passed8h 53m into the runtook 10m 5s
startup 173.0 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 173.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 13603, "content_end": "ctions, and the CPT theorem enforces the universal applicability of these symmetries, the correct answer is:\n\n\\boxed{A}\n"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 203.0 s, median 88.7 s; 8 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed11h 18m into the runtook 8m 15s
startup 148.8 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 148.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 224, "reasoning_characters": 1227, "content_end": "is less than the mean free path without the electron beam on. This aligns with opti ...[clipped]
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 170.4 s, median 44.0 s; 4 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Selfcheck #3: all gates passed — startup 148.8s, slowest 170.4s/median 44.0s (halved vs phase-2), 4 max_tokens, 0 timeouts, 32/32 well-formed. Submitting: