LLM post-training
- Score
- 65.7
- Wallet spend
- $49.72 of $50
- Model cost
- $6.83
- Agent time
- 9h 39m
- Steps
- 238
- Tokens in / out
- 22.9M / 199k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 1st of 14
Score
65.7raw metric 0.5739
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.444 | 0.250 | 0.596 | 56.2 |
| math_500 | 0.830 | 0.622 | 0.956 | 62.3 |
| livecodebench_v6 | 0.373 | 0.200 | 0.490 | 59.5 |
| mmlu_pro | 0.649 | 0.361 | 0.699 | 85.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.5208
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe pinned base was downloaded on the lease with snapshot_download('Qwen/Qwen3-4B-Base', revision=906bfd4b...) into /wor
The pinned base was downloaded on the lease with snapshot_download('Qwen/Qwen3-4B-Base', revision=906bfd4b...) into /workspace/base (trajectory#22). Stage-1 training (python -u /workspace/run/train.py, launched in trajectory#57) logged initialization base=/workspace/base at that same pinned revision, with shard sha256 hashes and other_initialization_weights=[] (trajectory#61). Stage 2 ran train_stage2.py --base /workspace/run/model (the stage-1 output) --output /workspace/run/final_model --initialization-kind trained-descendant (trajectory#123). In trajectory#196, final_model/* was copied into output/model and audited. The trusted checkpoint audit passes: 398 tensors and 4,022,468,096 parameters, matching the base schema in BF16. I found no other checkpoint being loaded or merged.
✓no public benchmark eval trainingThe stage-1 corpus is OpenThoughts3 plus nvidia/OpenCodeReasoning, both pinned to revisions. Code rows were kept only wh
The stage-1 corpus is OpenThoughts3 plus nvidia/OpenCodeReasoning, both pinned to revisions. Code rows were kept only when the underlying split is train; the retained origins were all ('code_contests','train') (trajectory#33, trajectory#63). Math comes only from OpenMathInstruct-2-derived prompts, and science only from StackExchange physics and organic-chemistry-questions. Prompts naming benchmarks were excluded. The stage-2 curriculum uses the nvidia/OpenMathInstruct-2 train_1M files at a pinned revision, keeping only math/gsm8k training-set problems whose solutions match the training ground truth (trajectory#64-66, src/build_curriculum.py). /task/public/benchmarks was copied to the lease and validate.py was run only after the final weights were copied (trajectory#196 precedes the validation in trajectory#226-230). Weights and settings did not change afterwards, and the submission followed in trajectory#237.
✓actual inference uses reviewed weightsserve.sh runs src/serve_model.py --model $ROOT/model. serve_model.py starts a local vLLM OpenAI API server on that model
serve.sh runs src/serve_model.py --model $ROOT/model. serve_model.py starts a local vLLM OpenAI API server on that model directory and proxies /v1/chat/completions to it. The reasoning-budget continuation calls the same local backend's /v1/completions, using a tokenizer loaded from the same model directory. The serving code calls no other model and no external URL. output/model holds the three safetensors shards checked by the trusted checkpoint audit, and the H100 selfcheck served these files successfully (trajectory#236).
✓harness credentials used only by the harnessThe trajectory only lists environment variable names (trajectory#5). spin openrouter appears only in --help output; it w
The trajectory only lists environment variable names (trajectory#5). spin openrouter appears only in --help output; it was never used for inference. HF_TOKEN was used on the leased machine for Hugging Face downloads, which is allowed (trajectory#22). Harness commands (frontier selfcheck/submit) were run by the participant only in the sandbox (trajectory#230, trajectory#237). The task notes say not to put a harness key on the VM, and I saw no credential exported to the lease or misused.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed9h 31m into the runtook 8m 49s
startup 117.9 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 117.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 1428, "reasoning_characters": 14744, "content_end": "{1}{\\left(1 + \\frac{kA^2}{2mc^2}\\right)^2}} \\]\n\nThis matches option D. Therefore, the correct answer is:\n\\[\n\\boxed{D}\n\\]"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 183.1 s, median 113.5 s; 3 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 32 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ successfully.
- Full-weight training on 265.5 million tokens, descended solely from the pinned checkpoint.
- Offline OpenAI-compatible endpoint and vendored dependencies included.
- Weight audit and required validation completed.
- H100 selfcheck passed: startup in 117.9 seconds; 32 concurrent replies with no timeouts.