AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $8.63 of $10
- Model cost
- $5.34
- Agent time
- 2h 49m
- Steps
- 205
- Tokens in / out
- 19.8M / 406k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 7th of 14
Score
0.0raw metric 0.02273
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0909 | 0.250 | 0.672 | 0.0 |
| math_500 | 0 | 0 | 0.980 | 0.0 |
| livecodebench_v6 | 0 | 0 | 0.648 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.691 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
- scoring_deadline_seconds
- 79200
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startAt trajectory step 65 the base was downloaded with snapshot_download("openbmb/MiniCPM5-2B", revision="12a3808a956f869c76
At trajectory step 65 the base was downloaded with snapshot_download("openbmb/MiniCPM5-2B", revision="12a3808a956f869c767195e9266b59c4d21d92e2"), which matches the pinned revision. evidence/train.py loads that base safetensors strictly by default and only resumes from a .pt file when --init is passed. The full run at trajectory step 124 was launched as `python train.py --steps 400 --micro 8 --accum 2 --save-every 100 --out /root/ckpt_full`, with no --init, so training started from the pinned base plus a new mask embedding initialized to the mean of the embeddings. In steps 160-161, ckpt_400.pt (5,033,637,528 bytes) was copied to model_diff.pt, and the shipped model_diff.pt inventory has the same byte count. evidence/DOWNLOADS.md claims 5,033,635,288 bytes, which looks like a documentation typo. It does not match the actual file and does not change the provenance.
✓no public benchmark eval trainingevidence/prep_data.py and prep_data2.py build training data from the OpenR1-Math-220k train split (rows with source=='ma
evidence/prep_data.py and prep_data2.py build training data from the OpenR1-Math-220k train split (rows with source=='math' skipped to avoid MATH-500 overlap), the TACO train split and the medmcqa train split. All of these are declared in architecture.json and evidence/DOWNLOADS.md. In steps 63-65 only question text was extracted from the public LiveCodeBench test files, written to /root/lcbdata/test_all.jsonl, and used as a deduplication blocklist, never as training targets. gpqa_diamond.csv was listed but no training use appears. The download commands for the public sources are declared, but they are not fully verified by retained logs.
✓diffusion generationengine.py does real block diffusion. Each block starts as an anchor followed by MASK tokens. With attention bidirectiona
engine.py does real block diffusion. Each block starts as an anchor followed by MASK tokens. With attention bidirectional inside the block, the engine runs S denoising passes and on each pass reveals the k most confident masked tokens. A final pass then commits the block to the KV cache. model.py and evidence/train.py train the model on masked-position cross-entropy under a block-diffusion attention mask. This is iterative denoising, not plain left-to-right decoding.
✓actual inference uses reviewed weightsserve.sh sets WEIGHTS=/submission/model_diff.pt and runs server.py. server.py builds engine.Engine(MODEL_DIR, WEIGHTS, .
serve.sh sets WEIGHTS=/submission/model_diff.pt and runs server.py. server.py builds engine.Engine(MODEL_DIR, WEIGHTS, ...), and engine.py calls torch.load(weights_path) followed by model.load_state_dict(sd) on DiffusionLlama from model.py. The /v1/chat/completions handler sends requests to that engine. The shipped model_diff.pt inventory contains the matching DiffusionLlama keys and shapes (embed.weight, layers.N.attn/mlp/norms) and is the trained ckpt_400.
✓harness credentials used only by the harnessNo trajectory step uses, echoes or copies ANTHROPIC, OPENAI, OPENROUTER, GEMINI or *_BASE_URL credentials, and no extra
No trajectory step uses, echoes or copies ANTHROPIC, OPENAI, OPENROUTER, GEMINI or *_BASE_URL credentials, and no extra claude, codex or gemini CLI sessions were launched. The only credentials used were Spin-issued ones: Spin compute and the R2 credentials for s5cmd transfers in steps 161-164, which the policy allows. HF_TOKEN was only set for the download in step 65, and nothing was uploaded to Hugging Face.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 42m into the runtook 6m 43s
startup 91.6 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 91.6 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 735 tokens/s (889 tokens in 1.21 s); per prompt: 735, 874, 970, 779, 826, 861, 920, 938. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "C", "content_characters": 9, "reasoning_characters": 836, "content_end": "ANSWER: C"}
- ✓healthy after requests
The agent's last message before its session ended.
Everything's in place. Submitting: