AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $34.56 of $50
- Model cost
- $54.53
- Agent time
- 9h 2m
- Steps
- 461
- Tokens in / out
- 137.5M / 381k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 9th of 14
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Verifier details
- failure
- throughput
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json names openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2, which matches the poli
architecture.json names openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2, which matches the policy. Training starts from base weights: training/train_diffusion.py loads LlamaDiffusion.from_pretrained(--base /models/minicpm5-2b). training/logs/train2.log shows the base loaded from /models/minicpm5-2b, and the only resume is from the run's own /data/ckpt2/ckpt_00400. The run ends at ckpt_01200. model/model.safetensors has the same BF16 tensor structure as the base. Caveat: the spin machine got its copy through `hf download openbmb/MiniCPM5-2B` with no --revision (trajectory#49, #50), and the participant never checked those weights against the operator-provided /artifacts/minicpm5-2b. The exact base revision used for training is therefore unverified. Still, the download is from the same repo and nothing shows it differs from the pinned commit. No other checkpoint was downloaded, so there is no established violation. The shipped tokenizer files were copied from /artifacts/minicpm5-2b.
✓no public benchmark eval trainingEvery training corpus is declared in manifest.md, and the code loads only train splits. training/prep_data.py calls load
Every training corpus is declared in manifest.md, and the code loads only train splits. training/prep_data.py calls load_dataset with split="train" and reads the OpenCodeReasoning split_0 train parquets. training/gen_star.py builds STaR data by self-generating answers to train-split questions from mmlu auxiliary_train, sciq, arc, obqa, csqa and race, and keeps only correct answers. No download logs were retained, so these declared public sources are unverified, but all are declared. The trajectory touches /task/public/benchmarks only for evaluation: a livecodebench line count (trajectory#14-37 chunk), and GPQA validation and inspection through validate.py (trajectory#441, #442). None of it was used as training data.
✓diffusion generationserve.sh launches server.py, where every request goes through Engine. Engine prefills the prompt, then generates in 128-
serve.sh launches server.py, where every request goes through Engine. Engine prefills the prompt, then generates in 128-token blocks that start as all-MASK (id 130082). Each denoising step runs one forward pass over the whole block with bidirectional attention (model.py, is_causal=False). _apply_logits then commits several positions in parallel: the run of positions above the confidence threshold tau, plus enough leftmost positions to finish within max_steps (6 or 12 per request after trajectory#455). Finished blocks are KV-cached. This is semi-autoregressive iterative denoising that commits many tokens per forward pass, not one-token-at-a-time left-to-right decoding. training/logs/server.log shows the CUDA-graph denoise stepper serving requests.
✓actual inference uses reviewed weightsserve.sh runs server.py --model $SUB/model, and server.py loads LlamaDiffusion.from_pretrained(args.model). In model.py,
serve.sh runs server.py --model $SUB/model, and server.py loads LlamaDiffusion.from_pretrained(args.model). In model.py, from_pretrained reads model/model.safetensors (5.03 GB, BF16, MiniCPM5-2B tensor layout per the weight inventory) and model/config.json. No other weights are fetched at serving time. In trajectory#454 the participant compared sha256 hashes of the shipped model.safetensors and /data/ckpt2/ckpt_01200, the final train2 checkpoint.
✓harness credentials used only by the harnessIn the trajectory, HF_TOKEN is used for HF downloads on the spin machine (allowed), and spin compute is used for trainin
In the trajectory, HF_TOKEN is used for HF downloads on the spin machine (allowed), and spin compute is used for training and serving (allowed). Searches for ANTHROPIC, OPENROUTER/openrouter keys, API_KEY and printenv found no misuse. The only OpenRouter references are the harness's own agent model metadata and the spin CLI's help text. No step uses harness credentials for the participant's own purposes.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed7h 38m into the runtook 4m 11s
startup 73.7 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 73.7 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 317 tokens/s (254 tokens in 0.80 s); per prompt: 317. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 2failed7h 47m into the runtook 4m 6s
startup 88.5 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 88.5 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 391 tokens/s (328 tokens in 0.84 s); per prompt: 391. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 3failed7h 57m into the runtook 3m 46s
startup 92.7 s of 300 s; 0/1 gates passed; failed single-stream speed: 2 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 92.7 s
- ✗single-stream speed · 2 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 425 tokens/s (613 tokens in 1.44 s); per prompt: 462, 425. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 4failed8h 38m into the runtook 3m 36s
startup 86.0 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 86.0 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 447 tokens/s (237 tokens in 0.53 s); per prompt: 447. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
The agent's last message before its session ended.
Both machines stopped. Now let me submit.