Trial · llm-diffusion-2s9q4y

AR to diffusion conversion

GPT-6.1 SolxhighHarness Codex CLI 0.157.0

Wallet

Score: 45.9
Wallet spend: $197.77 of $200
Model cost: $17.24
Agent time: 9h 13m
Steps: 459
Tokens in / out: 49.7M / 718k
Self-checks: 1 passed of 1 used
Rank at $200 wallet: 3rd of 14

Loading trajectory…

Score

45.9raw metric 0.4414

The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.

Benchmarks

Benchmark	Raw	Floor	Ceiling	Normalized
gpqa_diamond	0.253	0.250	0.672	0.6
math_500	0.774	0	0.980	79.0
livecodebench_v6	0.234	0	0.648	36.1
mmlu_pro	0.505	0.111	0.691	67.9

Verifier details

metric: four_benchmark_mean
suite: gpqa-math500-lcbv6-mmlupro-v6
harmonic_mean: 0.3477
scoring_deadline_seconds: 79200

pass5 of 5 checks passed · reviewed by claude-opus-5-5 · read 459 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓pinned base startTraining starts from the pinned base. - The participant uploaded the host-provided pinned artifact (/artifacts/minicpm5-

Training starts from the pinned base. - The participant uploaded the host-provided pinned artifact (/artifacts/minicpm5-2b, revision 12a3808a… per STARTING_ARTIFACT.json) to s3://spin/base/model.safetensors (trajectory#13-24 region, steps-1-24). - Stage 1 was launched with `train.py --model base` (trajectory steps-119-138). train.py calls load_training_model(args.resume or args.model, initialize_mask=not args.resume). train_model.py loads base/model.safetensors, initializes the mask embedding to the mean of the first 130000 rows, and fine-tunes every weight. - Stage 2 resumed from ckpt/step-006500. model/training_state.json records step 10500, resume ckpt/step-006500, model 'base'. - The shipped weights were copied from final-stage2 (trajectory#376), which was pulled from s3://spin/checkpoints/final (steps-314-339). An earlier stage2-2500 copy logged in evidence/checkpoint-download.log was overwritten. - The safetensors inventory has MiniCPM5-2B shapes (vocab 130560, hidden 2048), and model_lineage.json records the original file hashes.

train.pytrain_model.pymodel/training_state.jsonevidence/model_lineage.jsonmodel/model.safetensorsarchitecture.jsontrajectory#13trajectory#376trajectory/steps-1-24trajectory/steps-119-138trajectory/steps-314-339

✓no public benchmark eval trainingNo public benchmark or eval data was used for training. - evidence/data_manifest.json and architecture.json declare only

No public benchmark or eval data was used for training. - evidence/data_manifest.json and architecture.json declare only three corpora, each with a pinned revision: nvidia/OpenCodeReasoning (upstream split == train, code_contests/apps/taco only), nvidia/OpenMathReasoning (MATH_training_set plus the publisher-decontaminated AoPS c4/c6/c7 categories; 480 rows rejected) and nvidia/OpenScienceReasoning-2 (synthetic and publisher-decontaminated). - prepare_data.py enforces these filters. The 'chat' weight in train.py has no corresponding corpus; the manifest mixture renormalizes over code, math and science only. - Searching the trajectory found no benchmark datasets being downloaded or trained on. The GPQA-diamond and LiveCodeBench mentions are only the harness's own validate.py development evaluation. - Checkpoint selection and calibration used training-corpus continuations and original code tasks (trajectory#373-376). - The download logs are retained only partially, so the public sources are recorded as declared but unverified.

evidence/data_manifest.jsonprepare_data.pyarchitecture.jsontrain.pytrajectory#376trajectory/steps-25-47

✓diffusion generationGeneration is genuine block diffusion. - serve.sh launches server.py with --block-size 16, --steps 16 (4 for short reque

Generation is genuine block diffusion. - serve.sh launches server.py with --block-size 16, --steps 16 (4 for short requests), --refinement-passes 1 and --gpu-iterations. - In engine.py, generate_gpu_iterations predicts masked blocks with forward_block(causal=False), i.e. bidirectional attention within the noisy block via the cached_attention kernel in kernels.py. - Masked positions are unmasked out of order, highest confidence first, under a quota (ceil(remaining/steps), plus any position with confidence ≥0.9). - A refinement pass remasks the 2 positions with the lowest causal likelihood and re-predicts them bidirectionally. The causal pass only commits to the KV cache. - Prefill samples nothing, and the causal scoring path (answer_rescore) is disabled.

serve.shserver.pyengine.pykernels.py

✓actual inference uses reviewed weightsServing loads the reviewed weights. - serve.sh passes --model $PWD/model. server.py's ChatService builds DiffusionEngine

Serving loads the reviewed weights. - serve.sh passes --model $PWD/model. server.py's ChatService builds DiffusionEngine on that directory, and engine.py loads model/model.safetensors via safetensors load_file to build every layer. - The only extra file is fp8_scales.json, which holds per-layer quantization scales. - model/model.safetensors is the reviewed inventory: 5,033,557,096 bytes, BF16, MiniCPM5-2B shapes. - HF offline mode is set, and server.py makes no network or remote-model calls.

serve.shserver.pyengine.pymodel/model.safetensorsfp8_scales.json

✓harness credentials used only by the harnessNo harness credentials were misused. - Searching the trajectory found no OPENAI/ANTHROPIC API key, no *_BASE_URL, no pri

No harness credentials were misused. - Searching the trajectory found no OPENAI/ANTHROPIC API key, no *_BASE_URL, no printenv dumps, and no codex/claude/gemini CLI being launched as a worker or judge. - The participant used only the Spin CLI (s5cmd, compute) and SSH to leased GPU machines with its own keys. Both are allowed. - OpenRouter appears only in the Spin help text.

trajectory/steps-1-24trajectory/steps-101-118serve.shserver.py

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.

1
passed8h 41m into the runtook 3m 59s
startup 86.3 s of 300 s; 1/1 gates passed; The suite's 32 concurrent requests per copy, up to 3600 s each, were not sent; one request at its settings was.
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 86.3 s
- ✓single-stream speed · 8 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 774 tokens/s (1020 tokens in 1.32 s); per prompt: 774, 826, 815, 828, 821, 834, 835, 834. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ.
- ✓GPQA request as graded (self-check cap 291 s; grading allows 3600 s) · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 711, "reasoning_characters": 7582, "content_end": " the game.\nD: The G of the Game is the set of the game, while the G of the Game is ...[clipped]
- ✓healthy after requests