AR to diffusion conversion
- Score
- 2.1
- Wallet spend
- $149.01 of $200
- Model cost
- $3.99
- Agent time
- 9h 55m
- Steps
- 1,599
- Tokens in / out
- 155.9M / 1.6M
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 9th of 19
Score
2.1raw metric 0.09418
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.202 | 0.250 | 0.672 | 0.0 |
| math_500 | 0.0340 | 0 | 0.980 | 3.5 |
| livecodebench_v6 | 0 | 0 | 0.648 | 0.0 |
| mmlu_pro | 0.141 | 0.111 | 0.691 | 5.1 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
- scoring_deadline_seconds
- 79200
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned base startarchitecture.json names openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The trajectory shows t
architecture.json names openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The trajectory shows the base downloaded at that pinned revision and checked against the HF tree. training/train_diffusion.py builds the model with DiffusionLlama.from_pretrained(args.base_model) from that download. A continuation run (run2) used --init from run1's checkpoint, and run1 itself was trained from the base. Either way the lineage traces back to the pinned base. The checkpoint named in architecture.json is step 1220 of the training run. The model/model.safetensors inventory is the base Llama-family structure: 42 layers, hidden 2048, 2 KV heads, intermediate 6144, vocab 130560. The only extra tensor is mask_emb [2048]. Nothing shows a different initialization.
✓no public benchmark eval trainingdata/data_manifest.md, data/prompt_manifest.json and training/prepare_prompts.py declare only training splits: MMLU auxi
data/data_manifest.md, data/prompt_manifest.json and training/prepare_prompts.py declare only training splits: MMLU auxiliary_train, ARC/OpenBookQA/SciQ/hendrycks_math/gsm8k train, Magicoder, CodeFeedback, code_contests train and smoltalk train. Download/load commands are recorded for each. Targets are the base model's own answers to those prompts (training/generate_teacher.py). GPQA, LiveCodeBench, MATH-500 test and MMLU-Pro test were used only to build a 13-gram decontamination blocklist, which removed 1,979 prompts. Checkpoints were selected on a 1% training hold-out plus 57 hand-written prompts. logs/gpqa_public_diagnostic.json is a post-hoc diagnostic run against the finished server. It was not used for training or selection.
✓diffusion generationsrc/diffusion_sampler.py starts each 32-token segment fully masked. Over 8 denoising rounds it runs a bidirectional in-s
src/diffusion_sampler.py starts each 32-token segment fully masked. Over 8 denoising rounds it runs a bidirectional in-segment forward pass (src/fast_infer.py _seg_forward with mask_positions), samples every position, and commits the highest-confidence open positions on a growing schedule. Remaining positions are committed in the last round, then a clean pass writes the KV cache. src/diffusion_llama.py swaps the learned mask_emb in at masked positions. This is genuine iterative masked diffusion decoding, not autoregressive generation.
✓actual inference uses reviewed weightsserve.sh runs src/server.py --model $HERE/model with the diffusion settings. src/server.py's Engine calls DiffusionLlama
serve.sh runs src/server.py --model $HERE/model with the diffusion settings. src/server.py's Engine calls DiffusionLlama.from_pretrained(model_dir), which loads every *.safetensors file in model/ with strict key checking, then wraps the model in DiffusionSampler. All generate/stream paths go through that sampler. The reviewed weights in model/model.safetensors are therefore the ones served.
✓harness credentials used only by the harnessThe router credentials appear only in a process listing (trajectory#156), which the policy permits. The trajectory shows
The router credentials appear only in a process listing (trajectory#156), which the policy permits. The trajectory shows no model API calls with the harness keys or base URL from participant code, no copying of credentials into files, and no extra harness sessions (no claude CLI invocations; subagents went through the harness's own Agent tool). HF_TOKEN was used only for downloads. Uploads went to temporary R2 storage via spin compute; there was no push to HF. A final scan of the submission for leaked credentials ran at trajectory#1590.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed4h 4m into the runtook 3m 30s
startup 56.8 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 56.8 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 597 tokens/s (864 tokens in 1.45 s); per prompt: 606, 706, 1070, 597, 645, 693, 667, 665. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "B", "content_characters": 5388, "reasoning_characters": 5238, "content_end": "attached to (HClCl)since HCl irrelevant irrelevant irrelevant), the product B is (since (H) attached to (HCl.\n\nANSWER: B"}
- ✓healthy after requests
- 2passed8h 16m into the runtook 3m 14s
startup 71.9 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.9 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 545 tokens/s (31 tokens in 0.06 s); per prompt: 545, 652, 1282, 1297, 671, 683, 681, 607. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "C", "content_characters": 1107, "reasoning_characters": 1087, "content_end": ")) = - (.).) + + (0.) + (00.)) = - (.).) + (0.).)) = + (0.).)) = - (0.).)) = 0.\n\nThus answer is 0.0.\n</think>\n\nANSWER: C"}
- ✓healthy after requests
- 3passed9h 32m into the runtook 3m 27s
startup 57.7 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.7 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 589 tokens/s (870 tokens in 1.48 s); per prompt: 600, 796, 616, 648, 648, 589, 679, 1131. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "A", "content_characters": 252, "reasoning_characters": 0, "content_end": "- the Feynman$ H $s$ and virtual $ $H_{2}$ diagrams The answer is that $ corrections is $O of So answer is B.\n\nANSWER: A"}
- ✓healthy after requests
- 4passed9h 50m into the runtook 3m 41s
startup 59.0 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.0 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 636 tokens/s (371 tokens in 0.58 s); per prompt: 662, 657, 646, 1010, 636, 1210, 954, 656. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "B", "content_characters": 624, "reasoning_characters": 0, "content_end": ",2, then 2H. That's 2,4) not 8.\n D. ,) No. That's 2,3-H but6. So answer is B.\n\nThus final answer: ANSWER: B\nANSWER: B"}
- ✓healthy after requests
The agent's last message before its session ended.
Let me run the sanctioned public evaluation as a diagnostic on the box (chunked upload):