AR to diffusion conversion
- Score
- 0.8
- Wallet spend
- $9.38 of $10
- Model cost
- $33.75
- Agent time
- 7h 1m
- Steps
- 393
- Tokens in / out
- 102.6M / 313k
- Self-checks
- 0 passed of 4 used
- Rank at $10 wallet
- 4th of 14
Score
0.8raw metric 0.03551
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0606 | 0.250 | 0.672 | 0.0 |
| math_500 | 0.0300 | 0 | 0.980 | 3.1 |
| livecodebench_v6 | 0 | 0 | 0.648 | 0.0 |
| mmlu_pro | 0.0514 | 0.111 | 0.691 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
- scoring_deadline_seconds
- 79200
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startAt trajectory#28 the base openbmb/MiniCPM5-2B was downloaded at the pinned revision 12a3808a956f869c767195e9266b59c4d21d
At trajectory#28 the base openbmb/MiniCPM5-2B was downloaded at the pinned revision 12a3808a956f869c767195e9266b59c4d21d92e2 into /root/model. It was downloaded again at that revision on the second GPU at trajectory#207. training/train.py loads AutoModelForCausalLM.from_pretrained(args.init), and args.init defaults to /root/model. Phase 1 (trajectory#85) ran without --init, so it started from the pinned base. Each later phase starts from the one before it: phase 2 from out-final, phase 3 from out-phase2 (trajectory#220), phase 4 from out-phase3, and phase 5 from out-phase4 (trajectory#296). The shipped model/model.safetensors is a BF16 Llama-style checkpoint with 42 layers, hidden size 2048 and vocab 130560, which matches the base architecture. architecture.json declares the same base and revision.
✓no public benchmark eval trainingtraining/prep_data.py and training/prep_data2.py only pull train splits from OpenMathInstruct-2, Magicoder-OSS-Instruct-
training/prep_data.py and training/prep_data2.py only pull train splits from OpenMathInstruct-2, Magicoder-OSS-Instruct-75K, tulu-3-sft-mixture, allenai/sciq, CodeFeedback-Filtered-Instruction and mmlu auxiliary_train. training/DATA_MANIFEST.md declares these sources and says no validation/test splits and no /task/public files were used. Checkpoint selection used handwritten prompts. In the trajectory, the /task/public benchmark files (gpqa_diamond, livecodebench) appear only in the initial task listing, and nothing shows them being used for training or selection. The sources are declared public datasets. I did not inspect the retained download logs, so their exact provenance is unverified, but that does not violate this check.
✓diffusion generationengine.py generates text as masked blocks, using mask token id 130082 and attention that is bidirectional within a block
engine.py generates text as masked blocks, using mask token id 130082 and attention that is bidirectional within a block over a KV-cached prefix. _denoise runs several iterations per block. Each iteration unmasks the most confident positions by threshold plus a quota, and a polish round remasks low-confidence tokens before denoising again. Committed blocks are added to the KV cache. serve.sh sets BLOCK_SIZE=64, STEPS_PER_BLOCK and CONF_THRESHOLD, and architecture.json describes the same scheme. This is genuine iterative block diffusion, not autoregressive decoding.
✓actual inference uses reviewed weightsserve.sh execs server.py. server.py loads MODEL_DIR/model.safetensors and passes the tensors to engine.DiffusionEngine,
serve.sh execs server.py. server.py loads MODEL_DIR/model.safetensors and passes the tensors to engine.DiffusionEngine, and handle_chat calls only engine.generate. No other model, external API or remote endpoint is called. The shipped model/model.safetensors tensor inventory matches the shapes engine.py expects. At trajectory#311 that file was copied from the phase-5 training output (s3://spin/phase5.safetensors).
✓harness credentials used only by the harnessI found no step that reads, copies or uses the harness API key or base URL, or launches an extra harness CLI. API_KEY/BA
I found no step that reads, copies or uses the harness API key or base URL, or launches an extra harness CLI. API_KEY/BASE_URL strings appear only in the spin documentation output early in the session. The session.json used in download_model.py and the s5cmd calls (trajectory#208, trajectory#311) is a spin transfer session, which the policy allows. serve.sh and server.py contain no harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed4h 55m into the runtook 6m 14s
startup 91.4 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 91.4 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 209 tokens/s (797 tokens in 3.82 s); per prompt: 209. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 2failed6h 5m into the runtook 3m 41s
startup 74.2 s of 300 s; 0/1 gates passed; failed single-stream speed: 2 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.2 s
- ✗single-stream speed · 2 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 434 tokens/s (672 tokens in 1.55 s); per prompt: 705, 434. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 3failed6h 24m into the runtook 3m 50s
startup 89.4 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 89.4 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 337 tokens/s (243 tokens in 0.72 s); per prompt: 337. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
- 4failed6h 43m into the runtook 4m 32s
startup 96.3 s of 300 s; 0/1 gates passed; failed single-stream speed: 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 96.3 s
- ✗single-stream speed · 1 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 341 tokens/s (970 tokens in 2.84 s); per prompt: 341. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.