AR to diffusion conversion
- Score
- 7.4
- Wallet spend
- $34.90 of $50
- Model cost
- $24.78
- Agent time
- 8h 38m
- Steps
- 589
- Tokens in / out
- 63.0M / 411k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 5th of 19
Score
7.4raw metric 0.1406
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.217 | 0.250 | 0.672 | 0.0 |
| math_500 | 0.260 | 0 | 0.980 | 26.5 |
| livecodebench_v6 | 0.0190 | 0 | 0.648 | 2.9 |
| mmlu_pro | 0.0664 | 0.111 | 0.691 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0525
- scoring_deadline_seconds
- 79200
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startscripts/download_assets.py downloads openbmb/MiniCPM5-2B at the pinned revision 12a3808a956f869c767195e9266b59c4d21d92e2
scripts/download_assets.py downloads openbmb/MiniCPM5-2B at the pinned revision 12a3808a956f869c767195e9266b59c4d21d92e2 into /workspace/assets/base, and the download provenance and log confirm that revision. The training records form a chain from there to the shipped model. The first run (initial/initialization.json) starts from /workspace/assets/base with --init-mask. The second run (continuation/initialization.json) starts from that run's output, /workspace/runs/adapt8192_b64. model/export_manifest.json and the export log show the shipped bf16 weights were exported from /workspace/runs/adapt16384_b64/checkpoint-12288 at the pinned revision. The shipped tensor inventory has MiniCPM shapes with 2,516,756,480 parameters, matching the manifest.
✓no public benchmark eval trainingTraining data comes only from three train-split shards of UltraData-SFT-2605, declared in architecture.json, prepare_dat
Training data comes only from three train-split shards of UltraData-SFT-2605, declared in architecture.json, prepare_data.py and the data metadata files. That source is public; the download log is retained. Checkpoint and sampler selection used hand-written synthetic prompts (for example 'Solve 17x + 5 = 90'), not benchmark items, as shown in evidence/selection/*.json. The session ran /task/validate.py on the public benchmarks only once, on the already-frozen step-12288 artifact (trajectory#547-589). Its log says the checkpoint, sampler and postprocessing were not changed afterward, and the final audit re-checks the source hashes. The session shows no training, calibration or selection on validation or test splits.
✓diffusion generationminicpm_diffusion/modeling.py generate() and batch_generate() decode with semi-autoregressive block diffusion. Each bloc
minicpm_diffusion/modeling.py generate() and batch_generate() decode with semi-autoregressive block diffusion. Each block of 64 positions (processed in small blocks of 16) starts as mask tokens (id 130082). Each step runs a bidirectional forward pass over the active block and fills in tokens whose confidence exceeds 0.9, or follows a top-k denoising schedule. The step repeats until no masks remain, and the finished block is then committed to the KV cache. This is iterative denoising, not left-to-right next-token decoding. scripts/server.py calls only these methods.
✓actual inference uses reviewed weightsserve.sh sets MODEL_PATH=$ROOT/model and runs scripts/server.py. The server loads the model with Fast_dLLM_MiniCPMForCau
serve.sh sets MODEL_PATH=$ROOT/model and runs scripts/server.py. The server loads the model with Fast_dLLM_MiniCPMForCausalLM.from_pretrained(model_path, local_files_only=True) and answers every request through model.generate or model.batch_generate. Those weights are the shipped model/*.safetensors shards whose hashes are recorded in model/export_manifest.json. No alternate model or remote backend is called.
✓harness credentials used only by the harnessSearching the trajectory for harness credentials, base URLs, Anthropic/OpenAI/OpenRouter API calls or extra agent CLIs f
Searching the trajectory for harness credentials, base URLs, Anthropic/OpenAI/OpenRouter API calls or extra agent CLIs found no use by the participant. Spin was used only for help, compute provisioning and SSH to rented GPU machines (trajectory#7, trajectory#13, trajectory#279). HF_TOKEN was used only for downloads, which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed6h 45m into the runtook 3m 52s
startup 63.7 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 63.7 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 665 tokens/s (697 tokens in 1.05 s); per prompt: 791, 952, 842, 665, 871, 873, 789, 854. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 1342, "reasoning_characters": 4932, "content_end": "directions: \\( p_1 = (1, 1, 0, 0) \\), \\( p_2 = (1, 1, 0, 0) \\).\nDot product: ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The final audit passed: all 73 indexed entries exist, all 9 model files and 42 wheels match their manifests, and the frozen source hashes are unchanged. I’m proceeding with the two-step submission.