LLM post-training
- Score
- 0.0
- Wallet spend
- $9.90 of $10
- Model cost
- $1.39
- Agent time
- 8h 3m
- Steps
- 283
- Tokens in / out
- 26.7M / 402k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 10th of 14
Score
0.0raw metric 0.288
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.126 | 0.250 | 0.596 | 0.0 |
| math_500 | 0.600 | 0.622 | 0.956 | 0.0 |
| livecodebench_v6 | 0.0900 | 0.200 | 0.490 | 0.0 |
| mmlu_pro | 0.336 | 0.361 | 0.699 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.1690
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe training scripts download Qwen/Qwen3-4B-Base. src/run_b.sh does not pass an explicit --revision. Phase-1 LoRA (src/t
The training scripts download Qwen/Qwen3-4B-Base. src/run_b.sh does not pass an explicit --revision. Phase-1 LoRA (src/train_full.yaml) trains from that local base. Phase 2 (src/train_b4.yaml) continues that same adapter, and src/merge_b2.yaml merges it into merged_b2, which is uploaded as model2 (trajectory#179). assemble.sh copies model2 into the shipped model/ and patches only the eos/pad config (trajectory#180, #184). A tensor comparison in trajectory#254 shows embed_tokens identical to the base and small LoRA-sized deltas elsewhere. The trusted checkpoint audit reports 398 BF16 tensors whose parameter count equals the base. No other checkpoint was downloaded or merged. The missing revision pin alone does not establish a non-pinned start.
✓no public benchmark eval trainingsrc/prep.py and TRAINING_EVIDENCE.md declare public training-split sources: OpenThoughts-114k, DeepScaleR, Magicoder-Evo
src/prep.py and TRAINING_EVIDENCE.md declare public training-split sources: OpenThoughts-114k, DeepScaleR, Magicoder-Evol, CodeFeedback, Tulu-3, MMLU auxiliary_train and APPS train. Benchmark items (GPQA, LiveCodeBench, HumanEval, MBPP test/validation, AIME/AMC) are loaded only to build hash, prefix and title exclusion filters, and rows from banned benchmark sources are dropped. Sampling defaults were tuned on self-written prompts (src/pen_test.py, trajectory#253). Validation runs were measurement only. I saw no retained download logs, so the dataset downloads are unverified, but the sources are declared.
✓actual inference uses reviewed weightsserve.sh installs the vendored vLLM wheel and runs `vllm serve "$HERE/model"` on 127.0.0.1:8901. It then execs proxy.py,
serve.sh installs the vendored vLLM wheel and runs `vllm serve "$HERE/model"` on 127.0.0.1:8901. It then execs proxy.py, which only fills in missing sampling fields and forwards requests to that local upstream. The proxy makes no external API calls and does not alter responses. The served directory model/ contains the three inventoried safetensors shards that passed the trusted checkpoint audit.
✓harness credentials used only by the harnessSearching the trajectory found no use of ANTHROPIC credentials or copying of harness credentials. BASE_URL appears only
Searching the trajectory found no use of ANTHROPIC credentials or copying of harness credentials. BASE_URL appears only in Spin documentation output. HF_TOKEN was used only for Hugging Face downloads. The vendored anthropic wheel is not used by serve.sh or proxy.py, and the shipped serving path makes no LLM API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed5h 31m into the runtook 7m 22s
startup 128.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 128.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 30.7 s, median 29.9 s; 30 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed7h 51m into the runtook 5m 32s
startup 135.9 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 135.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 2481, "content_end": " answer is (3aR,4R,7S,7aS)-4,7-dimethyl-3a,4,7,7a-tetrahydro-4,7-epithioisobenzofuran-1,3-dione. \u0e40\u0e21\u0e37\u0e48\u0e2d\n\uf99b\n\n\u0e40\u0e23\u0e37\u0e48\nANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 33.6 s, median 12.5 s; 9 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed7h 58m into the runtook 5m 47s
startup 133.1 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 133.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 3025, "content_end": "he choices, \"Planet_1 is preferred due to its ~1.65 times higher probability to transit.\" is the correct one.\n\nANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 38.1 s, median 12.2 s; 5 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Sweep reveals the hard truth: constant EOS bias ≥3 stops generation at token 1 (greedy), and at temp 0.7 the model rambles into multilingual junk. Bias alone can't discriminate "answer complete". Testing penalty combos (frequency/presence penalty break repetition loops; mild bias terminates) — the classic remedy: