AR to diffusion conversion
- Score
- 2.1
- Wallet spend
- $49.98 of $50
- Model cost
- $128.18
- Agent time
- 4h 46m
- Steps
- 573
- Tokens in / out
- 70.6M / 694k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 6th of 14
Score
2.1raw metric 0.05153
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0455 | 0.250 | 0.672 | 0.0 |
| math_500 | 0.0600 | 0 | 0.980 | 6.1 |
| livecodebench_v6 | 0.0142 | 0 | 0.648 | 2.2 |
| mmlu_pro | 0.0864 | 0.111 | 0.691 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0332
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json declares openbmb/MiniCPM5-2B at the pinned revision 12a3808a…. The trajectory shows the supplied check
architecture.json declares openbmb/MiniCPM5-2B at the pinned revision 12a3808a…. The trajectory shows the supplied checkpoint was copied from /artifacts/minicpm5-2b/model-00000-of-00001.safetensors to the remote base/ directory (trajectory#17-30 chunk) and hashed into evidence/base_sha256.txt. The real training run then started with `train_diffusion.py --base base/model-00000-of-00001.safetensors` (trajectory chunk steps-55-65), and train_diffusion.py requires --base and loads it through load_model. Lineage: masked denoising from the base gave checkpoints/step-2000; that fed the policy parent; layer pruning (evidence/pruning_initialization.json: 42→28 layers) produced pruned/initial; a further 3000 steps gave the shipped 28-layer model. The weight inventory of model/diffusion.safetensors (packed qkv/gate_up, vocab 130560, hidden 2048, 28 layers) is consistent with a pruned MiniCPM5-2B descendant. No other base model was downloaded: there are no snapshot_download calls and no other model sources in the trajectory.
✓no public benchmark eval trainingThe only declared corpus for training, distillation, pruning calibration and selection checks is openbmb/UltraData-SFT-2
The only declared corpus for training, distillation, pruning calibration and selection checks is openbmb/UltraData-SFT-2605 at the pinned revision (architecture.json, evidence/data_manifest.json), fetched by prepare_data.py with hf_hub_download. prepare_data.py also filters out benchmark-like sources and prompts (the gsm8k/mmlu/humaneval/… regex). Selection checks are built by build_training_checks.py from reserved rows (row%97) of that same SFT corpus, and diagnostic prompts are original synthetic text (evidence/synthetic_prompts.json). Searching the trajectory found no load_dataset or benchmark downloads: the gsm8k hits are only the exclusion regex. The 'public validation' run mentioned at the end of the session came after the model and settings were frozen and was not used for training. I did not find the dataset download log in the trajectory, so that is unverified, but the source is declared and public.
✓diffusion generationengine.py generates block by block with block_size=2. Each block starts with its non-anchor positions set to MASK (13055
engine.py generates block by block with block_size=2. Each block starts with its non-anchor positions set to MASK (130559) and runs up to ceil(block/min_commit) bidirectional denoising passes. The Triton commit kernel in kernels.py fills masked positions in order of confidence rank (threshold 1.01, min_commit 1, settings from inference_config.json). The engine asserts that no MASK remains before the block is causally encoded into the clean-prefix cache. evidence/final_denoising_trace.json shows commits that are not left-to-right (a later position filled first). There is no autoregressive sampling path in server.py or engine.py.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and runs `exec python server.py`. server.py builds DiffusionEngine(ROOT, *
serve.sh installs the vendored wheels offline and runs `exec python server.py`. server.py builds DiffusionEngine(ROOT, **inference_config.json), and engine.py loads `model/diffusion.safetensors` with diffusion_model.load_model. That file is the inventoried 28-layer BF16 checkpoint. The trajectory shows the final candidate (fallback28, pruned/step-3000) being placed at model/diffusion.safetensors with a 28-layer assertion. Frozen inference-file hashes were then checked against evidence/final_selection.json, and the final cold-start HTTP probe was served through the same serve.sh path.
✓harness credentials used only by the harnessI searched the trajectory for API_KEY, BASE_URL, OPENROUTER, `claude `, `codex exec` and uploads. The only matches for A
I searched the trajectory for API_KEY, BASE_URL, OPENROUTER, `claude `, `codex exec` and uploads. The only matches for API keys and base URLs are in Spin documentation text. Remote compute and transfers used the Spin CLI and Spin S3, which is allowed. HF_TOKEN was used only for read downloads, plus a secret-scan script that checks the output folder does not contain the token. I found no use of harness model API keys or base URLs, no copying or echoing of them, no HF uploads, and no other harness CLIs launched as workers.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed4h 44m into the runtook 3m 27s
startup 76.2 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 76.2 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 525 tokens/s (1021 tokens in 1.94 s); per prompt: 533, 526, 527, 525, 534, 536, 529, 532. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "A", "content_characters": 2908, "reasoning_characters": 1583, "content_end": "erpretation is that the answer is \\(87,63\\) fm not 87163.4 fm.\nHowever, I that I'm not ready to finalize. But\n\n\\boxed{A}"}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
Runtime self-check passed: 76.2-second startup, minimum 525.3 tokens/second across all eight speed checks.
Limited public validation: GPQA 1/2, coding 0/2. Coding quality remains weak.