AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $34.52 of $200
- Model cost
- $13.54
- Agent time
- 9h 5m
- Steps
- 324
- Tokens in / out
- 70.2M / 285k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 9th of 14
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Verifier details
- failure
- throughput
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startTraining starts from the pinned base. The base was downloaded with `hf download openbmb/MiniCPM5-2B --revision 12a3808a9
Training starts from the pinned base. The base was downloaded with `hf download openbmb/MiniCPM5-2B --revision 12a3808a956f869c767195e9266b59c4d21d92e2 --local-dir /workspace/minicpm5-2b` (trajectory#31). This is also recorded in train/download.log and matches architecture.json. Stage 1, train_diffusion.py, loads /workspace/minicpm5-2b; its train.log header shows model=/workspace/minicpm5-2b, and it ran 3500 steps. Stage 2 (train_mcq.py) starts from hf_3500, the converted stage-1 checkpoint. Stage 3 (train_distill.py) starts from hf_mcq. convert_ckpt.py builds the model from the base config and loads each stage's model.pt. The final checkpoint is distill_ckpt/step-150, converted to hf_final (trajectory#291). The chain begins at the pinned base.
✓no public benchmark eval trainingNo public benchmark test or validation split was used for training. The declared corpora are fineweb-edu sample-10BT tra
No public benchmark test or validation split was used for training. The declared corpora are fineweb-edu sample-10BT train, cosmopedia web_samples_v2 train, MetaMathQA train and OpenCodeReasoning split_0 (stage 1), plus allenai/sciq train and cais/mmlu auxiliary_train (stages 2 and 3). All are train or auxiliary_train splits, declared in train/data_manifest.json and the training scripts. The stage 3 teacher prompts come only from sciq train and mmlu auxiliary_train (distill_batched.py); the stage 3 trainer reads only /workspace/distilled.json (trajectory#283). The HF 'Generating test/validation split' messages in the logs come from datasets materializing cais/mmlu, but the code takes only auxiliary_train. Five GPQA rows from the dev benchmarks went into quality_tests.json, but only as checkpoint quality probes (trajectory#114, trajectory#292), never as training data. Downloads of the declared public sources are logged in download.log.
✓diffusion generationdiffusion_generate.py implements block-wise masked diffusion. Each block starts fully masked with the mask token (130082
diffusion_generate.py implements block-wise masked diffusion. Each block starts fully masked with the mask token (130082). For steps_per_block iterations, it runs full bidirectional forward passes (is_causal=False), samples every masked position, and unmasks the top-k by confidence on a linear schedule. The MCQ variant runs the same diffusion reasoning, then predicts the answer letter at one masked position constrained to A–J. There is no autoregressive decoding path. The training objective is the MDLM/LLaDA masked-token cross-entropy with t~U(0,1).
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$SUBMISSION_DIR/model, runs offline, and starts uvicorn server:app. server.py loads AutoModelFor
serve.sh sets MODEL_DIR=$SUBMISSION_DIR/model, runs offline, and starts uvicorn server:app. server.py loads AutoModelForCausalLM from MODEL_DIR, forces is_causal=False on the config and every layer, and routes every chat completion through diffusion_generate or diffusion_generate_mcq, with no AR fallback. The shipped model/model.safetensors is 5033557128 bytes. That is exactly the size of hf_final/model.safetensors, which was uploaded and copied into output/model in trajectory#309 and trajectory#311, and hf_final came from the final distill checkpoint (trajectory#291). model/config.json is LlamaForCausalLM with is_causal false, and the inventory's tensor shapes match it.
✓harness credentials used only by the harnessNo harness credential misuse appears in the trajectory. HF_TOKEN was used only for HF downloads and dataset streaming on
No harness credential misuse appears in the trajectory. HF_TOKEN was used only for HF downloads and dataset streaming on the Spin GPU machine (trajectory#111, trajectory#288). Nothing was uploaded to HF. S3 transfers used Spin-issued transfer credentials (trajectory#307), which is permitted Spin tooling. I found no model-API calls with harness keys or base URLs, no copying of harness credentials, and no other harness CLIs launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed9h 1m into the runtook 4m 42s
startup 93.6 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 93.6 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 1012 tokens/s (120 tokens in 0.12 s); per prompt: 1082, 1037, 1012, 1078, 1054, 1058, 1024, 1049. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": "D", "content_characters": 331, "content_end": "1 1, 1, and 1, 0, is the sum of is the values of 0, C is 1, and 1.\nTherefore, B, 0, C, C, 185 is the total.\n\nANSWER: D"}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Converted MiniCPM5-2B into a masked text diffusion model with bidirectional attention that generates by iterative confidence-based denoising in 128-token blocks (4 steps each), served via an OpenAI-compatible endpoint.
Selfcheck passed: startup 93.6s (limit 300), single-stream speed 1011 tok/s (gate 450) on 8/8 prompts, GPQA-format reply correct shape. Submission in /workspace/output/ (85 files, 9.04 GB): diffusion weights, vendored offline wheels, serve.sh/server.py/diffusion_generate.py, architecture.json, and training evidence (3 stages, ~244M tokens, train/test splits excluded; MCQ format + teacher-distilled reasoning).