AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $45.00 of $50
- Model cost
- $20.38
- Agent time
- 10h 6m
- Steps
- 795
- Tokens in / out
- 76.3M / 843k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 9th of 14
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Verifier details
- failure
- throughput
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startIn the trajectory, the participant downloaded openbmb/MiniCPM5-2B at the pinned revision 12a3808a956f869c767195e9266b59c
In the trajectory, the participant downloaded openbmb/MiniCPM5-2B at the pinned revision 12a3808a956f869c767195e9266b59c4d21d92e2 with snapshot_download. training/train.py loads the base weights from --weights and only adds the MASK embedding row, set to the mean embedding. Run 2 (training/train_run2.log, 650 steps) saved out2/final, which was uploaded to R2 as final2.safetensors (trajectory#256, trajectory#257) and downloaded to weights_r2. At trajectory#300 it was copied into weights/model-00000-of-00001.safetensors; the md5 matched and the size (5,033,557,096 bytes) matches the weight inventory. The inventory shapes are consistent with the MiniCPM5-2B architecture: embed/lm_head [130560, 2048], kv [256, 2048], mlp [6144, 2048]. The fallback checkpoint went only to weights_fallback (trajectory#242). Run 3 never ran because its leases were rejected.
✓no public benchmark eval trainingThe training corpus is declared in architecture.json and training/data_manifest.json as openbmb/UltraData-SFT-2605, down
The training corpus is declared in architecture.json and training/data_manifest.json as openbmb/UltraData-SFT-2605, downloaded with hf_hub_download in training/prep_data.py. GPQA-Diamond and LiveCodeBench v6 were loaded only to build an 8-gram decontamination blocklist that drops overlapping examples (training/prep2.log: 266,631 blocklist entries). They were never used as training, calibration or selection data. The task's own validate.py was run once as a 3-example sanity probe (trajectory#342). That is evaluation with the provided tool, not training on or tuning against benchmark data. I found no derived or synthetic benchmark data.
✓diffusion generationThe model is trained with a masked-diffusion objective (training/train.py: random masking ratio t, with CE computed on m
The model is trained with a masked-diffusion objective (training/train.py: random masking ratio t, with CE computed on masked response tokens). At inference, graphs.py MultiRoundGraph and diffusion.py fill positions past the prompt with MASK tokens. Each step predicts k=4 masked positions in parallel from a single forward pass, with MASK logits excluded, and commits them. This is semi-autoregressive block diffusion: blocks are generated left to right, but tokens within each block are denoised in parallel. It is borderline because each block gets only one denoising step, but it is not AR decoding renamed. The eager fallback also has a draft+refine path.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py calls load_model(WEIGHTS), where WEIGHTS defaults to ./weights. model.py's load_model
serve.sh runs server.py. server.py calls load_model(WEIGHTS), where WEIGHTS defaults to ./weights. model.py's load_model/load_from read weights/model.safetensors.index.json and the shard weights/model-00000-of-00001.safetensors. server.py then builds a DiffusionSampler with MultiRoundGraph, falling back to RoundGraph and then eager mode, and /v1/chat/completions calls sampler.generate. The inventoried weights are the ones the server loads.
✓harness credentials used only by the harnessI searched the trajectory for harness API keys, base URLs, and claude/codex/gemini CLI launches and found none in partic
I searched the trajectory for harness API keys, base URLs, and claude/codex/gemini CLI launches and found none in participant code or commands. The AWS-style credentials in the lease setup are Spin-issued, session-scoped R2 transfer credentials for the Spin bucket (trajectory#257, trajectory#296), which is permitted Spin infrastructure. HF_TOKEN was used only for downloads; no uploads to Hugging Face were found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed3h 17m into the runtook 6m 23s
startup 92.8 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 92.8 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 135 tokens/s (1024 tokens in 7.58 s); per prompt: 135. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 2failed5h 42m into the runtook 4m 58s
startup 71.8 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.8 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 0 tokens/s (0 tokens in 30.03 s; TimeoutError: Submission HTTP request timed out after 30s); per prompt: 0. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 3failed6h 39m into the runtook 4m 39s
startup 77.6 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.6 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 59 tokens/s (1024 tokens in 17.43 s); per prompt: 59. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 4failed9h 53m into the runtook 3m 16s
startup 38.2 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 38.2 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 203 tokens/s (899 tokens in 4.44 s); per prompt: 203. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
The agent's last message before its session ended.
All files are in place, synced, and validated. Submitting: