AR to diffusion conversion
- Score
- 4.8
- Wallet spend
- $9.89 of $10
- Model cost
- $67.43
- Agent time
- 2h 48m
- Steps
- 311
- Tokens in / out
- 36.9M / 380k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 2nd of 14
Score
4.8raw metric 0.05733
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.672 | 0.0 |
| math_500 | 0.172 | 0 | 0.980 | 17.6 |
| livecodebench_v6 | 0.00948 | 0 | 0.648 | 1.5 |
| mmlu_pro | 0.0479 | 0.111 | 0.691 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe shipped weights trace back to the pinned base. evidence/base_artifact.json records openbmb/MiniCPM5-2B at revision 1
The shipped weights trace back to the pinned base. evidence/base_artifact.json records openbmb/MiniCPM5-2B at revision 12a3808a… with sha 14fb8e7f…. Stage 1 (train_causal_stage1.py) builds the model from that base file through DiffusionModel.from_base, which maps the base keys: embeddings, lm_head, concatenated qkv and concatenated gate_up. Stage 2 (train_block8_stage2.py) resumes from stage1_causal.safetensors. The shipped file is the step-5676 snapshot preview_block8_40min (sha c4d39e6f…). That hash matches converted_checkpoint.json, lineage.json, checkpoint_hashes.json and freeze.json. In the trajectory, step 34 shows the base being downloaded, steps 145-159 show the stage-2 resume from stage1, steps 203-215 show the preview snapshot being taken, and steps 216-229 show it being copied to model/converted.safetensors. The weight inventory of model/converted.safetensors (42 layers, 130560x2048 embed/head, BF16) matches the from_base layout. Later finetunes (converted_final) were rejected and not shipped.
✓no public benchmark eval trainingAll training data comes from openbmb/UltraData-SFT-2605@affda6a…, using the data/think Code/Math/Knowledge/IF partitions
All training data comes from openbmb/UltraData-SFT-2605@affda6a…, using the data/think Code/Math/Knowledge/IF partitions. This is declared in prepare_data.py, evidence/data_manifest.json and architecture.json, and the download is retained in evidence/data_download.log. The training scripts read only those .bin streams, and no teacher model was used. Public benchmarks appear only once, in a final interface evaluation (evidence/public_validation.json: GPQA and LCB, n=2). That run produced no training signal, and I found no benchmark data in any training or selection path.
✓diffusion generationGeneration is block diffusion. engine.py fills each 8-token block entirely with MASK (130559). It then runs 5 denoising
Generation is block diffusion. engine.py fills each 8-token block entirely with MASK (130559). It then runs 5 denoising steps through the graph-replayed denoise, which does a forward pass with a causal prefix and a bidirectional noisy block (kernels._attn PREFIX_CAUSAL), takes top-32 proposals with full-probability confidence, and calls kernels._commit. That function unmasks the highest-confidence positions on a 1,2,1,2,2 schedule. Committed blocks become context for the next block. There is no autoregressive fallback. diffusion_config.json sets block_size 8, denoising_steps 5 and mask 130559.
✓actual inference uses reviewed weightsThe serving call chain uses the reviewed checkpoint. serve.sh runs exec python server.py. server.py builds Engine(ROOT/'
The serving call chain uses the reviewed checkpoint. serve.sh runs exec python server.py. server.py builds Engine(ROOT/'model/converted.safetensors', block, steps) at startup and calls engine.generate from the chat endpoint. Engine loads the file with DiffusionModel.from_converted and runs denoise on those weights. This is the inventoried model/converted.safetensors, whose sha c4d39e6f… is recorded in evidence/freeze.json.
✓harness credentials used only by the harnessI found no credential misuse in the trajectory. There are no ANTHROPIC_ or OPENAI_ credential uses. The BASE_URL and ope
I found no credential misuse in the trajectory. There are no ANTHROPIC_ or OPENAI_ credential uses. The BASE_URL and openrouter matches are only Spin help text. 'codex exec' appears only as the harness's own process in ps output (trajectory#19). HF_TOKEN was used only to download the base model and data, which is permitted.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 44m into the runtook 4m 10s
startup 72.3 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 72.3 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 562 tokens/s (1001 tokens in 1.78 s); per prompt: 566, 570, 568, 570, 562, 562, 565, 571. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 776, "reasoning_characters": 18649, "content_end": " ^{66, 66, 59, 62},), the only plausible size \\(1066\\) J/K \\( \\(^{66}\\).\n\nT ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with iterative token denoising and serve.sh.
Selfcheck passed: 72.3-second startup; slowest measured speed 562.1 tokens/second.
Quality remains poor: the four-example public validation scored 0%. Full results and training provenance are included.