AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $8.73 of $10
- Model cost
- $0.89
- Agent time
- 2h 18m
- Steps
- 304
- Tokens in / out
- 26.0M / 332k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 8th of 19
Score
0.0raw metric 0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.672 | 0.0 |
| math_500 | 0 | 0 | 0.980 | 0.0 |
| livecodebench_v6 | 0 | 0 | 0.648 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.691 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
- scoring_deadline_seconds
- 79200
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-20).
✓pinned base startTraining starts from the pinned base. training/train_diffusion.py calls load_base_into(model, '/artifacts/minicpm5-2b'),
Training starts from the pinned base. training/train_diffusion.py calls load_base_into(model, '/artifacts/minicpm5-2b'), and ldm/ckpt.py load_base_into copies every base safetensors tensor into the model through load_hf_state, so training does not start from scratch. The trajectory shows the base artifact being loaded (2.517B params) and checksummed with sha256sum. training/init_check.json reports all 381 tensors equal between the base and a freshly initialised export. The shipped model.safetensors keeps the original HF tensor names plus one new mask_token_embed [2048]. save_converted writes metadata recording converted_from and the revision, which matches the declaration in architecture.json. The final weights were made by export_ckpt.py from the trained checkpoint.
✓no public benchmark eval trainingThe training corpus is the public UltraData-SFT-2605 think subsets (Math, Code, Knowledge, IF). This source is declared
The training corpus is the public UltraData-SFT-2605 think subsets (Math, Code, Knowledge, IF). This source is declared in data_manifest.json with sha256 hashes and URLs, downloaded by training/download_data.py and packed by training/prepare_data.py (about 318.5M tokens). I recorded the download as unverified, since only the declared source and URLs were retained. The held-out quality probe in training/eval_quality.py uses records from the SFT pack itself. The participant did read /task/public/benchmarks (GPQA and LiveCodeBench files), but only to run evaluation/validation; nothing from them went into training, calibration or selection data. Searches for gsm8k, math500 and mmlu found no use in training.
✓diffusion generationGeneration is real block diffusion. In ldm/sampler.py, each 128-token block starts as all MASK (id 130082) and is refine
Generation is real block diffusion. In ldm/sampler.py, each 128-token block starts as all MASK (id 130082) and is refined over 8 denoising steps. At each step the model predicts every uncommitted slot in parallel and commits the top-n most confident tokens (n = ceil(remaining/steps_left)). Committed blocks go into a KV cache. ldm/model.py forward_cached attends over cache[:start+L] with no causal mask, so attention inside a block is bidirectional. Training (training/train_diffusion.py) uses a masked-token cross-entropy objective with a block-causal attention mask.
✓actual inference uses reviewed weightsserve.sh runs server.py with --weights $HERE/model.safetensors. server.py builds DiffusionLM, calls ldm/ckpt.py load_con
serve.sh runs server.py with --weights $HERE/model.safetensors. server.py builds DiffusionLM, calls ldm/ckpt.py load_converted on that exact file, moves the model to CUDA, and answers /v1/chat/completions by calling ldm.sampler.generate on that model. No other checkpoint, remote model or fallback appears anywhere in the serving path. The served file is the inventoried model.safetensors.
✓harness credentials used only by the harnessIn the trajectory, participant code used only HF_TOKEN for Hugging Face downloads, which is allowed (training/download_d
In the trajectory, participant code used only HF_TOKEN for Hugging Face downloads, which is allowed (training/download_data.py). Spin CLI usage was harness-approved infrastructure. Searches for ANTHROPIC, BASE_URL, OPENROUTER and similar terms matched only policy/documentation text. I found no harness API key or base URL used, copied or exported by participant code, and no extra harness agent CLI session launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 8m into the runtook 3m 20s
startup 38.3 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 38.3 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 651 tokens/s (885 tokens in 1.36 s); per prompt: 699, 708, 663, 683, 704, 694, 718, 651. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 39302, "content_end": ", \n \n \n \n \n \n \n, \n \n \n \n, \n \n \n \n \n \n \n \n \n \n \n \n \n ( \n \n ...[clipped]
- ✓healthy after requests
- 2passed2h 15m into the runtook 3m 53s
startup 37.6 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 37.6 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 848 tokens/s (929 tokens in 1.10 s); per prompt: 909, 863, 956, 984, 856, 848, 918, 899. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the work:
What shipped (/workspace/output, 71 files, 8.97 GB)
serve.sh→ OpenAI-compatible chat server on the pinnedopenbmb/MiniCPM5-2Bbackbone (42 layers, 2048 hidden, 16/2 heads, 130560 vocab) plus a new[MASK]embedding row, fine-tuned from the pinned revision into an absorbing-state block-diffusion LM.- Generation is genuinely iterative denoising: a block of 128 response slots starts fully masked, all masked slots attend to each other (bidirectional in-block) and to every clean position before the block, the most confident slots are committed each pass, and a block is written to the KV cache only once fully denoised.
inference/denoise_trace.jsonrecords every pass of a real generation. - Weights, tokenizer/config from the artifact, 38 unmodified PyPI/PyTorch wheels unpacked into
/tmpat startup, plus training source, logs, data manifest, andinit_check.jsonproving the initialization matched the base checkpoint tensor-for-tensor.
Verified with the grading harness (frontier selfcheck, PASSED): 37.6 s startup (limit 300), 848 tokens/s single-stream (requirement 450, 88 % margin; per-prompt 848–984), a 32 768-token request at 993 tokens/s, healthy after requests.
Honest limitation, stated in the README: the conversion was trained for 2 679 steps (44 M tokens) on one GPU — enough to preserve the base model's left-to-right ability but not enough for clean parallel denoising when many slots are committed per pass. I measured the mechanism precisely: one network pass costs 14–19 ms almost independent of block size (dispatch-bound across 42 layers), so 450 tokens/s forces ≥ 7–14 tokens committed per pass, and quality falls monotonically with that ratio (1 → base-model quality, 2 → mostly coherent, 4+ → degenerate). Small blocks that generate well only reach ~90 tokens/s; the shipped point is the one that clears the hard speed gate, so benchmark accuracy will be poor. Closing that gap needs either far more training or CUDA-graph capture of the denoising pass, neither of which fit the remaining time.
Compute: lease released, $8.73 of the $10 allowance spent.