AR to diffusion conversion
- Score
- 0.4
- Wallet spend
- $125.64 of $200
- Model cost
- $35.80
- Agent time
- 10h 19m
- Steps
- 930
- Tokens in / out
- 89.8M / 580k
- Self-checks
- 1 passed of 4 used
- Rank at $200 wallet
- 11th of 19
Score
0.4raw metric 0.004
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.672 | 0.0 |
| math_500 | 0.0160 | 0 | 0.980 | 1.6 |
| livecodebench_v6 | 0 | 0 | 0.648 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.691 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
- scoring_deadline_seconds
- 79200
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base starttraining/download_assets.py and evidence/download_manifest.json download openbmb/MiniCPM5-2B at the pinned revision 12a3
training/download_assets.py and evidence/download_manifest.json download openbmb/MiniCPM5-2B at the pinned revision 12a3808a956f869c767195e9266b59c4d21d92e2, with sha256 hashes. training/train.py loads --base-model (/workspace/assets/base/model-00000-of-00001.safetensors) with a strict state-dict load. evidence/initialization-shifted.json records no missing or unexpected keys and 2,516,756,480 parameters. Every training launch in the trajectory, including the shifted run that produced the selected step-12000 checkpoint, uses that base file (trajectory#254, trajectory#151). The shipped model.safetensors inventory keeps the base tensor names in BF16 at nearly the same size as the base shard. The selected artifact is the original shifted step-12000 checkpoint, not a continuation (trajectory#894).
✓no public benchmark eval trainingThe only training corpus is 4 UltraData-SFT-2605 think shards at a pinned dataset revision, with hashes recorded in evid
The only training corpus is 4 UltraData-SFT-2605 think shards at a pinned dataset revision, with hashes recorded in evidence/download_manifest.json and training/download_assets.py. Download logs are retained in the trajectory (trajectory#80). training/preprocess.py and evidence/data_manifest.json sample 8460 train rows plus a 176-row train-derived holdout and explicitly exclude the /task/public/benchmarks files. Trajectory searches show benchmark files were only listed, plus the task's own validation code being read, and were excluded from preprocessing (trajectory#14, trajectory#42). I found no benchmark evaluation runs or uses of benchmark data in training or selection. Selection used the train-derived holdout and authored synthetic tests (TRAINING.md).
✓diffusion generationinference.py DiffusionEngine.generate starts each 32-token block fully masked with mask id 130559. It runs bidirectional
inference.py DiffusionEngine.generate starts each 32-token block fully masked with mask id 130559. It runs bidirectional attention within the active block over a causal clean-prefix KV cache and fills the block over 2 passes (short requests) or 16 passes (long requests). Each pass reveals the most confident still-masked positions in parallel, then the block is committed. diffusion_model.py builds the block mask, and config.json and architecture.json declare the absorbing masked-diffusion objective. This is iterative parallel denoising, not autoregressive one-token decoding.
✓actual inference uses reviewed weightsserve.sh execs server.py --model-dir $ROOT. server.py builds a DiffusionEngine and calls engine.generate for /v1/chat/co
serve.sh execs server.py --model-dir $ROOT. server.py builds a DiffusionEngine and calls engine.generate for /v1/chat/completions, with no other model and no external API calls. inference.py loads model_dir/model.safetensors into DiffusionForMaskedLM with load_state_dict(strict=True), and CUDA graphs replay those same modules. The trajectory shows the shipped model.safetensors (sha256 153e4b50...) was copied from the shifted step-12000 candidate (trajectory#893, trajectory#894), which matches TRAINING.md.
✓harness credentials used only by the harnessTrajectory searches found no ANTHROPIC or OPENAI API key use, no harness BASE_URL use, and no claude/codex CLI launches.
Trajectory searches found no ANTHROPIC or OPENAI API key use, no harness BASE_URL use, and no claude/codex CLI launches. The only BASE_URL matches are in Spin's own help text. Credential-related activity was limited to Spin compute and transfer sessions and an HF token file uploaded to the GPU node for hf_hub_download downloads (trajectory#80). I found no HF upload or push. None of the shipped code (server.py, inference.py, training/*) uses harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 0m into the runtook 3m 25s
startup 68.7 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.7 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 435 tokens/s (559 tokens in 1.29 s); per prompt: 435. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 2failed2h 23m into the runtook 4m 2s
startup 82.4 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 82.4 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 430 tokens/s (375 tokens in 0.87 s); per prompt: 430. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 3failed2h 35m into the runtook 3m 58s
startup 84.0 s of 300 s; 0/1 gates passed; failed single-stream speed: 3 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.0 s
- ✗single-stream speed · 3 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 449 tokens/s (398 tokens in 0.89 s); per prompt: 592, 595, 449. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 4passed10h 12m into the runtook 7m 11s
startup 71.5 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.5 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 1224 tokens/s (1000 tokens in 0.82 s); per prompt: 1224, 1347, 1405, 1378, 1378, 1386, 1325, 1242. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓healthy after requests
The agent's last message before its session ended.
The post-selfcheck artifact also passed all static and wheel-hash checks. The submission is final: 167 files, 8.99 GB, with the frozen shifted-logit step-12,000 checkpoint.