AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $6.98 of $50
- Model cost
- $21.80
- Agent time
- 2h 49m
- Steps
- 428
- Tokens in / out
- 1.6M / 745k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 11th of 19
The reviewer found a rule violation; see the Review tab.
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✗diffusion generation
Generation is not genuine iterative denoising; it is speculative decoding relabelled as diffusion. In src/bd_engine.py, _single_bd_step feeds the denoiser clean tokens plus mask-canvas tokens but uses only the hidden state at the last clean position (d_s1[:, 4:5]) and throws away the mask-position outputs. Draft tokens c2..c4 are then produced one after another, each from the previous token's embedding and hidden state under causal attention, with an optional n-gram lookup override. The 42-layer backbone verifies [c0..c4] causally and keeps the longest prefix whose draft log-prob is within log_alpha of the backbone's top-1, then appends the backbone's top-1 as a bonus token. generate_paged_batch does the same with threshold log(0.10). This is EAGLE-style left-to-right decoding, not denoising of a block.
✓pinned base startarchitecture.json declares openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The trajectory show
architecture.json declares openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The trajectory shows the base checkpoint is the operator-provided /artifacts/minicpm5-2b. Training logs report initializing from /artifacts/minicpm5-2b. The final conversion loads that source directory, writes the mask embedding rows 130500:130510, zeroes the unused lm_head rows and saves the result. The shipped backbone inventory is a full 42-layer MiniCPM-shaped checkpoint consistent with that path. I found no evidence of a different initialization.
✓no public benchmark eval trainingTraining data is openbmb/UltraData-SFT-2605 at a pinned revision, declared in architecture.json, data_manifest.json and
Training data is openbmb/UltraData-SFT-2605 at a pinned revision, declared in architecture.json, data_manifest.json and src/download_data.py, with a retained download log. That gives a 13500-sample train split and a 350-sample dev calibration set held out from UltraData. Benchmark files were used only for decontamination fingerprints, validation and speed-gate prompts (trajectory#357 saved GPQA/LCB prompts only for speed testing). I found no evidence of training or calibrating on benchmark data.
✓actual inference uses reviewed weightsserve.sh runs src/server.py with --model-dir pointing at the shipped model directory. server.py builds MiniCPM5BlockDiff
serve.sh runs src/server.py with --model-dir pointing at the shipped model directory. server.py builds MiniCPM5BlockDiffusionEngine from it, and bd_engine.py loads model/model-00000-of-00001.safetensors and model/denoiser.safetensors, both of which appear in the reviewed weight inventories. Both the single-stream and batch paths use these loaded weights.
✓harness credentials used only by the harnessTrajectory searches for API keys, BASE_URL, ANTHROPIC and Gemini found no use by the participant. The openrouter subcomm
Trajectory searches for API keys, BASE_URL, ANTHROPIC and Gemini found no use by the participant. The openrouter subcommand appears only in the Spin CLI help output, with no evidence it was ever called. The participant used the Spin CLI for compute and transfer and HF_TOKEN for dataset and model downloads. I found no evidence of harness credentials being used outside the harness.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 45m into the runtook 3m 39s
startup 25.2 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 25.2 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 536 tokens/s (1024 tokens in 1.91 s); per prompt: 616, 611, 587, 651, 564, 536, 747, 604. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "D", "content_characters": 813, "reasoning_characters": 18517, "content_end": "multiple particles. Thus, the spectrum remains continuous with an adjusted shape, and the endpoint increases.\n\nANSWER: D"}
- ✓healthy after requests
The agent's last message before its session ended.
Waiting for task-68 to complete.