AR to diffusion conversion
- Score
- 78.8
- Wallet spend
- $55.84 of $200
- Model cost
- $26.96
- Agent time
- 3h 35m
- Steps
- 545
- Tokens in / out
- — / —
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 1st of 19
Score
78.8raw metric 0.6333
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.460 | 0.250 | 0.672 | 49.7 |
| math_500 | 0.886 | 0 | 0.980 | 90.4 |
| livecodebench_v6 | 0.570 | 0 | 0.648 | 87.9 |
| mmlu_pro | 0.618 | 0.111 | 0.691 | 87.3 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.5989
- scoring_deadline_seconds
- 79200
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json declares openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The task's provided
architecture.json declares openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The task's provided artifact at /artifacts/minicpm5-2b records that revision (trajectory#1). It was copied to /dev/shm/minicpm5-2b on the GPU node with s5cmd (trajectory#65, #69). train_block_diffusion.py loads the teacher and the v1 student from that directory; the v1 init record shows base_model_path=/dev/shm/minicpm5-2b. The v2 run continued from the v1 output (--init-student-path /dev/shm/minicpm5-2b-diffusion, trajectory#388). In the inventory of model/model.safetensors (381 tensors, BF16, 42 layers, vocab 130560) every tensor shape matches the MiniCPM5-2B config. The only additions are mask tokens trained into existing embedding rows.
✓no public benchmark eval trainingThe training data loader reads only the openbmb/UltraData-SFT-2605 think shards (math, code, knowledge and instruction-f
The training data loader reads only the openbmb/UltraData-SFT-2605 think shards (math, code, knowledge and instruction-following) at a pinned revision. download_data.py, data_manifest.json (with sha256 hashes) and download_data.log record these downloads, and architecture.json says no calibration or selection set was used. GPQA-diamond and LiveCodeBench files were copied to the GPU node (trajectory#65, #67). They were used only as HTTP prompts for speed-gate checks and with validate.py for evaluation (trajectory#268, #406, #410). The trajectory shows no training, calibration or selection on them.
✓diffusion generationdiffusion_engine.py generates each block as an anchor followed by 9 position-offset MASK tokens. The first denoising pas
diffusion_engine.py generates each block as an anchor followed by 9 position-offset MASK tokens. The first denoising pass is bidirectional (is_causal=False) and unmasks positions by confidence: those at or above the threshold, and at least the top 5. A second bidirectional pass refines the partly unmasked block. A causal verify/commit pass then writes the KV cache and accepts a prefix. Tokens therefore come from iterative masked denoising, not plain autoregressive decoding. model/config.json declares mask_token_id 130082, 16 mask tokens and model_paradigm block_diffusion, and the training script trains the model with block-diffusion masked objectives.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR to $SCRIPT_DIR/model and execs server.py. server.py builds BlockDiffusionEngine(MODEL_DIR, ...),
serve.sh sets MODEL_DIR to $SCRIPT_DIR/model and execs server.py. server.py builds BlockDiffusionEngine(MODEL_DIR, ...), and _load_weights in diffusion_engine.py loads every model/*.safetensors into the tensors used for prefill and denoising. In the trajectory, /dev/shm/minicpm5-2b-diffusion-v2/model.safetensors was uploaded to transfer storage and then downloaded to /workspace/output/model/model.safetensors (trajectory#463, #466, #469). Its 5,033,557,128-byte size matches the reviewed inventory, and the trajectory also checks it (trajectory#471).
✓harness credentials used only by the harnessThe trajectory runs on the antigravity-cli/Gemini harness. Searches of the code and trajectory found no API_KEY, BASE_UR
The trajectory runs on the antigravity-cli/Gemini harness. Searches of the code and trajectory found no API_KEY, BASE_URL or harness credential references, no extra launches of Gemini, Claude or Codex CLIs, and no reads of the harness config directory. The participant used Spin CLI compute and s5cmd, plus temporary S3 transfer credentials issued by `spin transfer credentials` to move data (trajectory#62, #65). These are Spin storage credentials meant for VMs, not harness model credentials. The shipped serving and training code makes no harness or OpenRouter calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed3h 14m into the runtook 3m 51s
startup 68.6 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.6 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 515 tokens/s (1024 tokens in 1.99 s); per prompt: 534, 587, 528, 515, 585, 535, 552, 573. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "D", "content_characters": 28652, "content_end": "s a ketone? C + HNO2 -> D. HNO2 can do\n</think>\n\nTherefore, based on the analysis above, the final answer is:\n\nANSWER: D"}
- ✓healthy after requests
- 2passed3h 19m into the runtook 3m 58s
startup 71.2 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.2 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 485 tokens/s (1024 tokens in 2.11 s); per prompt: 493, 642, 554, 542, 589, 485, 525, 549. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "B", "content_characters": 20599, "content_end": ", an ester with para-substitution, matching all data. The NMR data are consistent with ethyl 4-aminobenzoate.\n\nANSWER: B"}
- ✓healthy after requests
- 3passed3h 25m into the runtook 3m 59s
startup 73.9 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 73.9 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 457 tokens/s (1024 tokens in 2.24 s); per prompt: 506, 634, 496, 540, 570, 457, 552, 543. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "C", "content_characters": 31674, "content_end": "A? \"Ten-fold dilution is more than 3.3\n</think>\n\nTherefore, based on the analysis above, the final answer is:\n\nANSWER: C"}
- ✓healthy after requests
- 4passed3h 31m into the runtook 3m 21s
startup 60.4 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 60.4 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 478 tokens/s (1023 tokens in 2.14 s); per prompt: 537, 501, 493, 499, 520, 478, 478, 603. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "A", "content_characters": 22979, "content_end": "No, hex-2-ynoate is CH3CH2CH2COOCH=CH?\n</think>\n\nTherefore, based on the analysis above, the final answer is:\n\nANSWER: A"}
- ✓healthy after requests