AR to diffusion conversion
- Score
- 0.0
- Wallet spend
- $65.69 of $200
- Model cost
- $16.27
- Agent time
- 12h 1m
- Steps
- 758
- Tokens in / out
- 66.9M / 874k
- Self-checks
- 0 passed of 1 used
- Rank at $200 wallet
- 9th of 14
The agent ran out of its 12 hours; the verifier graded what was in /workspace/output.
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original autoregressive MiniCPM5-2B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Verifier details
- failure
- throughput
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned base startarchitecture.json names openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2, and the session downlo
architecture.json names openbmb/MiniCPM5-2B at revision 12a3808a956f869c767195e9266b59c4d21d92e2, and the session downloaded that pinned snapshot. The final training run at trajectory#485 started from the snapshot: it passes `--ckpt $snap` from the HF cache for models--openbmb--MiniCPM5-2B. train.py loads it with load_ar_checkpoint in code/diffllm/model.py. That function maps the HF keys and adds a single new mask_emb vector, initialized to the mean embedding. The shipped model/model.safetensors fits this lineage: it has DiffLlama key names, BF16 tensors and a base-identical config, and it is about 1.7KB larger than the base shard, which matches the size of the [2048] mask_emb tensor. That run finished at trajectory#564. Its output was pushed as model_final at trajectory#565 and copied to output/model/model.safetensors.
✓no public benchmark eval trainingmake_prompts.py and make_train.py build training prompts and gold-label filters only from training splits: MATH train, G
make_prompts.py and make_train.py build training prompts and gold-label filters only from training splits: MATH train, GSM8K train, MMLU auxiliary_train, MBPP train and CodeContests train. These are declared in evidence/data_manifest.md and were downloaded at trajectory#42. The answers are self-distilled from the base model. Benchmark sets are used only in decontam.py, to drop overlapping rows with 8-gram shingles: GPQA, LiveCodeBench v6, MMLU-Pro test and MATH test. evidence/decontam_log.txt records 'kept 45395 dropped 2890'. The final run at trajectory#485 trained on /root/data/train_clean.jsonl.gz, the decontaminated set. MMLU-Pro test was fetched at trajectory#62 only for decontamination. The GPQA 25-question file is a validation-only evaluation. One unverified detail: the manifest gives the CodeContests repo as google-research-datasets, while the session used google-deepmind; both are the public training split. No benchmark evaluation data entered training.
✓diffusion generationcode/server.py generates by block diffusion. Each 128-token block starts fully masked with MASK_ID/mask_emb. denoise_blo
code/server.py generates by block diffusion. Each 128-token block starts fully masked with MASK_ID/mask_emb. denoise_block runs 8 iterative bidirectional window_forward passes and commits tokens in order of confidence, with Gumbel sampling. run_blocks moves left to right and re-encodes the committed prefix before each block. Matching masked-diffusion training is implemented in code/datagen_sources/train.py: masked cross-entropy on response tokens, with prefix-LM attention in code/diffllm/model.py. Generation is not autoregressive next-token decoding.
✓actual inference uses reviewed weightsThe serving chain is: serve.sh installs the vendored wheels and runs exec python3 code/server.py. env_paths in server.py
The serving chain is: serve.sh installs the vendored wheels and runs exec python3 code/server.py. env_paths in server.py resolves model/model.safetensors, model/model_config.json and model/tokenizer.json. Engine then calls load_diffusion_model in code/diffllm/model.py, a strict loader that raises on missing or unexpected keys. All decoding runs through that loaded DiffLlama. No external model API or alternate checkpoint is called. Hashes at trajectory#559-560 confirm that the served files match the submission.
✓harness credentials used only by the harnessI searched the trajectory for ANTHROPIC, OPENROUTER, OPENAI_API_KEY, API_KEY and BASE_URL. The only matches were Spin CL
I searched the trajectory for ANTHROPIC, OPENROUTER, OPENAI_API_KEY, API_KEY and BASE_URL. The only matches were Spin CLI help text in trajectory steps 11-33. Remote work used Spin compute/exec/s5cmd and HF downloads, which the policy permits. No step shows the participant using harness credentials outside the harness.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed11h 43m into the runtook 19m 27s
startup 89.1 s of 300 s; 0/1 gates passed; failed single-stream speed: 2 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 89.1 s
- ✗single-stream speed · 2 of 8 measured prompts, sent one at a time after an untimed warm-up (max_tokens 1024, temperature 1), each reply's reasoning and answer tokens counted with the pinned openbmb/MiniCPM5-2B tokenizer over end-to-end request time (sandbox clock), a miss re-sent once; slowest 251 tokens/s (658 tokens in 2.62 s); per prompt: 713, 251. Public prompts: 7 GPQA and 2 hard LiveCodeBench; grading draws 4 GPQA, 3 level-5 MATH-500 (held out) and 2 hard LiveCodeBench, so its replies differ. Grading stops here with score 0.
The agent's last message before its session ended.
Selfcheck failed — getting details: