AR to diffusion conversion
- Score
- 2.1
- Wallet spend
- $38.39 of $50
- Model cost
- $3.25
- Agent time
- 10h 47m
- Steps
- 1,774
- Tokens in / out
- 256.5M / 769k
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 5th of 14
Score
2.1raw metric 0.04705
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.00505 | 0.250 | 0.672 | 0.0 |
| math_500 | 0.0560 | 0 | 0.980 | 5.7 |
| livecodebench_v6 | 0 | 0 | 0.648 | 0.0 |
| mmlu_pro | 0.127 | 0.111 | 0.691 | 2.8 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json and evidence/initialization.json declare openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b5
architecture.json and evidence/initialization.json declare openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2. Trajectory step 7 shows STARTING_ARTIFACT.json naming the same repo and revision. At step 844 the training base was fetched with snapshot_download at that pinned revision, and the shipped model file was also uploaded to the remote base directory. The shipped model/model-00000-of-00001.safetensors was copied from /artifacts/minicpm5-2b (trajectory steps 1-19). Its inventory is BF16 Llama tensors whose shapes match the base config: vocab 130560, hidden 2048, 42 layers. train_diffusion.py keeps the base frozen and trains only a rank-64 LoRA and a mask embedding. Phase 1 started from the base and phase 2 (the selected adapter) continued from phase 1 via --adapter-init.
✓no public benchmark eval trainingtrain_diffusion.py, evidence/data_manifest.json and evidence/download_recipe.txt declare only public training splits: Hu
train_diffusion.py, evidence/data_manifest.json and evidence/download_recipe.txt declare only public training splits: HuggingFaceH4/ultrachat_200k train_sft, CodeAlpaca_20K train and gsm8k main/train. Neither GPQA nor LiveCodeBench appears in them. The trajectory shows the benchmark files under /task/public/benchmarks only in directory listings. They were never loaded into training or selection. Model selection (evidence/model_selection_results.txt) used five handwritten prompts. evidence/validation_gpqa_smoke.json records a single evaluation-only smoke run that was not used for training or selection. There is no retained download log for the public corpora, so their provenance is declared but unverified.
✓diffusion generationGeneration in serve.py goes through DenoisingEngine.denoise_block, which works in 256-token spans. Each span starts as p
Generation in serve.py goes through DenoisingEngine.denoise_block, which works in 256-token spans. Each span starts as prefix + [MASK]*length (MASK id 130100). On each pass the model predicts every masked position, and the highest-confidence positions are committed, with everything remaining committed on the final pass. Passes per span: 4 for requests up to 1024 tokens; for longer requests, 16 on the first span and 4 on later spans. In modeling_diffusion.py the forward pass applies causal attention over the prompt prefix and bidirectional attention over the response span. This is iterative masked diffusion decoding, not autoregressive token-by-token generation.
✓actual inference uses reviewed weightsserve.sh executes /submission/serve.py directly, with HF_HUB_OFFLINE=1. serve.py calls DiffusionLM.from_checkpoint on /s
serve.sh executes /submission/serve.py directly, with HF_HUB_OFFLINE=1. serve.py calls DiffusionLM.from_checkpoint on /submission/model with model/diffusion_adapter.safetensors. In modeling_diffusion.py that call loads model/model-00000-of-00001.safetensors and then the adapter's mask_embedding and LoRA tensors. Both files appear in the weight inventory with matching shapes: rank-64 F32 LoRA tensors for all 42 layers plus the mask embedding. The server makes no external API calls and runs no other model.
✓harness credentials used only by the harnessNo trajectory step shows the participant's code using, copying or echoing harness API credentials. Searches for OPENAI_A
No trajectory step shows the participant's code using, copying or echoing harness API credentials. Searches for OPENAI_API_KEY and OPENROUTER found no hits, BASE_URL appears only in Spin documentation text, and no extra harness CLI (such as codex exec) was launched as a worker. The read-only HF_TOKEN was used only to download the base model and datasets, which the task explicitly permits.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed4h 28m into the runtook 3m 53s
startup 69.4 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.4 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 857 tokens/s (607 tokens in 0.71 s); per prompt: 857, 1402, 1334, 1160, 1298, 1374, 1233, 1412. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 2775, "reasoning_characters": 0, "content_end": "lylylylylylylylylylyl.yl theylylylylylylylylylylylylylylylylylylylylylylylylAlyl well ...[clipped]
- ✓healthy after requests
- 2failed4h 35m into the runtook 3m 46s
startup 57.4 s of 300 s; 0/1 gates passed; failed single-stream speed: 5 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.4 s
- ✗single-stream speed · 5 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 186 tokens/s (342 tokens in 1.83 s); per prompt: 541, 473, 516, 455, 186. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 3passed10h 44m into the runtook 4m 33s
startup 70.5 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.5 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 883 tokens/s (797 tokens in 0.90 s); per prompt: 959, 1035, 960, 1010, 973, 1081, 883, 960. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 438, "reasoning_characters": 0, "content_end": " ... ... .. .. ... .. ... ... ... ... ... .. .. .. .. .... ... .. .. .. .. ... .. .. . ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Selfcheck passed: startup 70.5s, and all eight speed prompts passed with the slowest at 883 tokens/s. The bounded GPQA item was wrong, but I did not use it to select the adapter. I’m submitting the frozen folder now.