AR to diffusion conversion
- Score
- 80.9
- Wallet spend
- $38.05 of $50
- Model cost
- $124.00
- Agent time
- 9h 29m
- Steps
- 337
- Tokens in / out
- 133.6M / 514k
- Self-checks
- 2 passed of 4 used
- Rank at $50 wallet
- 1st of 14
Score
80.9raw metric 0.64
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.545 | 0.250 | 0.672 | 70.1 |
| math_500 | 0.942 | 0 | 0.980 | 96.1 |
| livecodebench_v6 | 0.456 | 0 | 0.648 | 70.3 |
| mmlu_pro | 0.616 | 0.111 | 0.691 | 87.1 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.5961
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startThe starting checkpoint was the provided /artifacts/minicpm5-2b/ artifact. Its STARTING_ARTIFACT.json names openbmb/Mini
The starting checkpoint was the provided /artifacts/minicpm5-2b/ artifact. Its STARTING_ARTIFACT.json names openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2 (trajectory#1, trajectory#3). At trajectory#151 the participant copied /artifacts/minicpm5-2b/model-00000-of-00001.safetensors directly to model/model.safetensors. The inventory of model/model.safetensors matches the MiniCPM5-2B structure: lm_head and embed_tokens are [130560, 2048], q/o projections [2048, 2048], k/v projections [256, 2048], and the MLP is 6144 wide. architecture.json declares this file as the unchanged base. All trained parameters are in a separate file, model/denoiser_lora.safetensors, which holds rank-128 lora.{i}_{q,k,v,o,gate,up,down}.A/B tensors plus a 2048-dim mask_embedding. These were trained on top of the frozen base. training/download_data.sh also downloads the pinned revision.
✓no public benchmark eval trainingThe training corpora are all declared in architecture.json and training/download_data.sh, and all come from train splits
The training corpora are all declared in architecture.json and training/download_data.sh, and all come from train splits: UltraData-RL-2609 (Math, Knowledge), codeparrot/apps train, ai2_arc ARC-Challenge train, sciq train, and UltraData-SFT-2605 think shards. The prompts for self-distillation (training/build_prompts.py) and the dev set (training/build_devset.py) are built only from these train splits, and the training code does not reference GPQA. The trajectory shows /task/public/benchmarks used only through validate.py. One point needs noting. After a GPQA-Diamond validation run (trajectory#307-#312), the participant looked at unparseable outputs, which included degenerate loops and truncated answers. In response they added a generic decoding fix: control tokens are committed only at the first masked position, and an answer_threshold of 0.98 applies after </think> (trajectory#313-#314). This reused the existing 0.98 code threshold. A GPQA rerun was killed at trajectory#315, before it produced a score, and the fix was evaluated and adopted on the training-split dev set (trajectory#317-#327, training/logs/dev_eval_summary.txt). After that, GPQA and LCB were run for reporting only (trajectory#328-#335). No training, fine-tuning, distillation or parameter calibration on public benchmark inputs, targets or rewrites is established. The public download sources are declared; this review did not independently verify them beyond the shipped download script and logs.
✓diffusion generationdllm/engine.py implements masked block diffusion with blocks of 16 tokens and [MASK] id 130082 (the mask embedding comes
dllm/engine.py implements masked block diffusion with blocks of 16 tokens and [MASK] id 130082 (the mask embedding comes from denoiser_lora). Each denoising step runs bidirectional attention over the noisy block, and attends causally to committed context through the KV cache. The denoiser LoRA is applied only to noisy-block rows. All positions whose confidence exceeds the threshold (0.9, or 0.98 in code and the final answer) are committed in parallel, in any order. Each step also samples the first masked position so it always makes progress. The block advances once it is fully unmasked. Measured tokens per step are about 1.4-1.7 in the dev logs and server logs (trajectory#330, training/logs/dev_eval_summary.txt). This shows genuine parallel, iterative unmasking, not autoregressive decoding renamed. An exact-AR debug mode exists only behind DLLM_DEBUG_AR, which serve.sh does not set.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$HERE/model and HF_HUB_OFFLINE=1, then runs uvicorn dllm.server:app. dllm/server.py builds Engin
serve.sh sets MODEL_DIR=$HERE/model and HF_HUB_OFFLINE=1, then runs uvicorn dllm.server:app. dllm/server.py builds Engine(MODEL_DIR) using model/diffusion_config.json. Engine._load loads model/model.safetensors (every *.safetensors except the LoRA file) and model/denoiser_lora.safetensors, including the lora.* A/B tensors and mask_embedding. It then builds merged int8 W+BA weights for fast steps from those same tensors. The scheduler loop runs prefill, step and advance on this engine. The weight inventories contain exactly the keys the loader expects. The final selfcheck at trajectory#330 served from this layout.
✓harness credentials used only by the harnessIn the trajectory, the participant used Spin to rent compute (vast.ai leases, extend and stop) and the frontier selfchec
In the trajectory, the participant used Spin to rent compute (vast.ai leases, extend and stop) and the frontier selfcheck/submit commands. Searches for ANTHROPIC, API_KEY, BASE_URL and 'claude -p' turned up only Spin documentation. No harness credential was found being used directly by the participant for model calls or other out-of-policy purposes.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 39m into the runtook 3m 31s
startup 38.8 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 38.8 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 233 tokens/s (1022 tokens in 4.40 s); per prompt: 233. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 2failed6h 15m into the runtook 3m 4s
startup 31.7 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 31.7 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 448 tokens/s (1021 tokens in 2.28 s); per prompt: 448. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 3passed8h 6m into the runtook 3m 4s
startup 39.8 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 39.8 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 451 tokens/s (1021 tokens in 2.27 s); per prompt: 499, 451, 501, 527, 510, 566, 483, 697. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "C", "content_characters": 195, "reasoning_characters": 13746, "content_end": "ch is greater than the acceptable 0.3 cycles. This deviation between technical replicates is the discrepancy.\n\nANSWER: C"}
- ✓healthy after requests
- 4passed9h 4m into the runtook 3m 58s
startup 41.7 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 41.7 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 491 tokens/s (1022 tokens in 2.08 s); per prompt: 526, 565, 491, 552, 544, 554, 553, 526. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "D", "content_characters": 561, "reasoning_characters": 16495, "content_end": "ngements and do not show optical isomerism. Thus, compounds 1 and 2 are the ones that show optical isomerism.\n\nANSWER: D"}
- ✓healthy after requests
The agent's last message before its session ended.
LCB v6 sample (first 20 problems, reporting only): 50% (10/20). Everything is in place and unchanged since the passing selfcheck, so I'm handing it in.