AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $38.39 of $200
- Model cost
- $63.18
- Agent time
- 4h 14m
- Steps
- 283
- Tokens in / out
- 98.8M / 238k
- Self-checks
- 0 passed of 2 used
- Rank at $200 wallet
- 9th of 14
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Verifier details
- failure
- throughput
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json declares openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The diff-v1 trainin
architecture.json declares openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. The diff-v1 training metadata pulled from the remote run (trajectory#172) records the same base and pinned revision, and the starting artifact /artifacts/minicpm5-2b matches it. The shipped weights are diff-v1/checkpoint-250, downloaded at trajectory#170-171, and their size matches the model/model.safetensors inventory. model/config.json and the tensor shapes match the 42-layer base architecture. The remaining base files were copied from /artifacts at trajectory#180. Nothing shows a different initialization.
✓no public benchmark eval trainingdownload_data.py, download_commands.txt and logs/data_manifest.json declare the only training corpus: training shards of
download_data.py, download_commands.txt and logs/data_manifest.json declare the only training corpus: training shards of openbmb/UltraData-SFT-2605. train_diffusion.py also filters out benchmark-named substrings. The /task/public/benchmarks directory (gpqa_diamond.csv, livecodebench-v6) was only listed at trajectory#121. No step read, loaded or trained on those files.
✓diffusion generationserver.py denoise() starts each 32-token block as all MASK tokens (id 130082) and refines it over several forward passes
server.py denoise() starts each 32-token block as all MASK tokens (id 130082) and refines it over several forward passes. Attention inside the block is bidirectional (all-zero additive mask). Each step commits in parallel every token above a 0.5 confidence threshold, with at least one commit per step. The KV cache is updated only from clean blocks, so generation is block-causal across blocks. complete() calls denoise for every request, so this is real iterative masked diffusion decoding, not autoregressive sampling. train_diffusion.py trains the matching masked-diffusion objective.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR to ./model and runs server.py. server.py load() calls AutoModelForCausalLM.from_pretrained(MODEL
serve.sh sets MODEL_DIR to ./model and runs server.py. server.py load() calls AutoModelForCausalLM.from_pretrained(MODEL_DIR), which loads the reviewed model/model.safetensors and model/config.json. The /v1/chat/completions handler calls complete(), which calls denoise() with that loaded model. No other weights or remote endpoints are used.
✓harness credentials used only by the harnessSearches of the trajectory and the shipped code found no use of harness API keys or base URLs (for example API_KEY, BASE
Searches of the trajectory and the shipped code found no use of harness API keys or base URLs (for example API_KEY, BASE_URL, XAI or OpenRouter). The only credential used was HF_TOKEN, to download the public dataset and the base model (download_data.py). The Spin compute tooling used only its own CLI auth. serve.sh uses no external credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed3h 49m into the runtook 4m 13s
startup 61.8 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 61.8 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 41 tokens/s (662 tokens in 16.02 s); per prompt: 41. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 2failed4h 6m into the runtook 5m 1s
startup 72.8 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 72.8 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 26 tokens/s (526 tokens in 20.42 s); per prompt: 26. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
The agent's last message before its session ended.
The MiniCPM5-2B checkpoint is now a block diffusion model, and the submission is in.
Generation fills a 32-token block with an absorbing mask, scores the whole block at once with bidirectional attention inside the block, and commits the confident tokens before rescoring. The prefix cache is updated only after the block is clean, so this is iterative denoising rather than left-to-right decoding. serve.sh installs the vendored wheels and serves POST /v1/chat/completions as submission.
On the grading image the server was healthy in 62 seconds, inside the 300-second limit. The speed gate measured about 41 tokens per second against the 450 requirement. The adapted model only trusts about one or two tokens per score, and a full 42-layer score stays launch-bound at several milliseconds, so forcing more commits per step destroyed the text. The shipped checkpoint is the one that still writes a real answer and stops.