LLM post-training
- Score
- 48.3
- Wallet spend
- $49.61 of $50
- Model cost
- $90.92
- Agent time
- 7h 8m
- Steps
- 575
- Tokens in / out
- 63.8M / 301k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 3rd of 14
Score
48.3raw metric 0.5188
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.460 | 0.250 | 0.596 | 60.6 |
| math_500 | 0.684 | 0.622 | 0.956 | 18.6 |
| livecodebench_v6 | 0.293 | 0.200 | 0.490 | 32.0 |
| mmlu_pro | 0.639 | 0.361 | 0.699 | 82.1 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.4641
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startTraining started from the pinned base. On the GPU host, trajectory#36 shows hf_hub_download pulling Qwen/Qwen3-4B-Base a
Training started from the pinned base. On the GPU host, trajectory#36 shows hf_hub_download pulling Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539 into /workspace/base. Stage 1 (run_training.sh, then src/train.py) loads it with AutoModelForCausalLM.from_pretrained(..., local_files_only=True). The initial hashes logged in evidence/training.jsonl match evidence/pinned_base_hashes.json and trajectory#37. Stage 2 (src/train_phase2.py, launched with --base /workspace/result/model) starts from the stage 1 output: the initial hashes in evidence/phase2_training.jsonl equal the stage 1 final hashes in architecture.json and evidence/stage1_final_audit.json. src/finalize_training_evidence.py checks that lineage and that no other weight sources exist. It then installs stage2_final/model as output/model (trajectory#414–#417). src/install_model.py only audits the checkpoint, writes the template and docs, and moves it into place; it merges no foreign weights. The trusted checkpoint audit confirms 3 BF16 shards with 398 tensors and the same parameter count as the Qwen3-4B-Base revision.
✓no public benchmark eval trainingevidence/data_manifest.json, src/prepare_data.py and src/prepare_phase2.py use only declared public TRAIN splits: - Open
evidence/data_manifest.json, src/prepare_data.py and src/prepare_phase2.py use only declared public TRAIN splits: - OpenThoughts3-1.2M train - OpenMathInstruct-2 train_1M - Mixture-of-Thoughts science train - codeforces-cots, filtered to code_contests TRAIN IDs - code_contests train - MBPP train Stage 2 adds format data derived from those TRAIN sources plus replay. No source code reads GPQA or LiveCodeBench files. The trajectory touches /task/public/benchmarks only to count records and plan the post-freeze validate.py run (trajectory/steps-301-378, step ~304–305). Public validation results were produced after the freeze and used only for reporting (README.md, architecture.json, evidence/public_validation_summary.json). Retained download logs for the public sources were not separately checked line by line, so they are recorded as unverified; this does not affect the verdict.
✓actual inference uses reviewed weightsserve.sh runs the vLLM OpenAI api_server with --model "$ROOT/model" and --generation-config "$ROOT/model", and installs
serve.sh runs the vLLM OpenAI api_server with --model "$ROOT/model" and --generation-config "$ROOT/model", and installs wheels offline with no external API calls. The served directory holds exactly model-00001..00003-of-00003.safetensors, which the trusted checkpoint inventory audit passes (BF16, base layout, untied_head false). src/reasoning_budget.py is a local logits processor that only forces </think> after a token budget. It does no network access and loads no other model.
✓harness credentials used only by the harnessSearching the trajectory for ANTHROPIC, OPENAI, BASE_URL and API_KEY found no participant use of harness credentials. Th
Searching the trajectory for ANTHROPIC, OPENAI, BASE_URL and API_KEY found no participant use of harness credentials. The only matches are Spin documentation and the validator's argparse source, printed while inspecting validate.py (the --judge-key-env MLE_JUDGE_API_KEY default, in trajectory/steps-475-510). No judge key was supplied or used. Remote compute and transfers went through Spin (spin s5cmd), and the HF download in trajectory#36 is allowed. serve.sh makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed20m 50s into the runtook 12m 19s
startup 172.3 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 172.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 1654, "content_end": "-methylbenzenesulfonohydrazide.\n\nAnswer: A. N'-(3-(hydroxymethyl)-5-isopropylcyclohexyl)-4-methylbenzenesulfonohydrazide"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 224.8 s, median 9.0 s; 1 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 2passed46m 34s into the runtook 8m 38s
startup 177.9 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 177.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 1720, "content_end": "it_1 / P_transit_2 = 1.65\n\nTherefore, Planet_1 has a ~1.65 times higher probability to transit than Planet_2.\n\nAnswer: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 142.2 s, median 6.3 s; 4 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
- 3passed7h 0m into the runtook 9m 41s
startup 166.5 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 166.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "B", "content_characters": 11, "reasoning_characters": 12769, "content_end": "\n\nANSWER: B"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 219.8 s, median 121.7 s; 11 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The final self-check passed: startup took 166.5 seconds, all 32 replies were valid, and the slowest took 219.8 seconds. Eleven replies reached the token limit; none timed out. I’m submitting the unchanged package now.