All tasks
Task · speculative-drafting

Speculative drafting

Train a small draft model that speeds up greedy speculative decoding for Qwen3-8B

EfficiencyInspired byGoogle Research
Score
mean accepted length
Best
48.4
Median over trials
8.4

Instruction

Train a draft model for speculative decoding with Qwen3-8B as the target, run in non-thinking mode with greedy decoding. The target's tokenizer and configuration are in /artifacts/qwen3-8b/, and /task/public/TARGET.json names its revision. I have provided 40 example conversations with the target's greedy continuations.

You can start from any open-weights model that isn't a specialized speculative drafting model.

Write a script serve.sh which serves the drafter. It should have a POST /draft endpoint which takes the token IDs of a conversation so far and returns up to 16 token IDs that it predicts the target will generate next. Run offline on one H100, ready within 300 seconds. The whole submission, including weights, code and dependencies, must be at most 1,000,000,000 bytes. The held-out set is 200 conversations, five times the public examples; each request must answer within 60 seconds, and answering all of its requests may take at most 10,800 seconds in total.

Optimize the mean acceptance length.

Do not train, fine-tune, calibrate or select on the validation or test split of any public dataset or benchmark; training splits and other public corpora are fine.

Submit in /workspace/output/. Interface and runtime details: /task/CONTRACT.md. Supporting files: /task/public/. Evaluate with python /task/validate.py --server-url URL. Test the finished submission the way it will be run with frontier selfcheck. Hand it in with frontier submit when it is final. Use spin --help for compute and model APIs.

Trials

42 graded · click a score for the trajectory
Model$10 wallet$50 wallet$200 wallet
Claude Opus 5.5xhigh21.237.041.8
Claude Sonnet 5.5xhigh20.833.848.4
Claude Fable 5.1xhigh15.626.829.5
Claude Haiku 5.5max5.518.621.3
GPT-6 Astraxhigh16.816.525.8
GPT-6.1 Solxhigh8.10.025.2
GPT-6 Lunaxhigh0.00.00.0
Gemini 3.8 Flashhigh0.00.00.0
Grok 4.7xhigh0.00.00.0
Muse Spark 1.3max0.00.08.7
Muse Wondermax9.513.15.2
Kimi K3max0.09.90.0
Qwen3.8 Maxxhigh0.00.07.0
GLM 5.3max0.012.013.7

Scoring

The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.

MetricScores 0 atScores 100 at
mean_acceptance_length4.67