Speculative drafting
- Score
- 6.6
- Wallet spend
- $12.75 of $50
- Model cost
- $1.00
- Agent time
- 7h 23m
- Steps
- 466
- Tokens in / out
- 43.4M / 355k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 12th of 19
Score
6.6raw metric 0.06617
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.7588
- tokens
- 44314
- steps
- 9312
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.5832
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json and evidence/DATA_MANIFEST.md declare every corpus. The sources are prompts generated by an LLM throug
architecture.json and evidence/DATA_MANIFEST.md declare every corpus. The sources are prompts generated by an LLM through the participant's own spin proxy (evidence/gen_prompts.py), synthetic templates, and the public training splits of ultrachat train_sft, alpaca and dolly (evidence/fetch_prompts.py and evidence/download_commands.txt). Targets are greedy Qwen3-8B continuations of those prompts (evidence/gen_target.py). Selection and GPTQ calibration use the participant's own holdouts and training splits (refs_p2_train, dev2/dev3, refs_p2_hold). The shipped public validation.jsonl was only replayed for local agreement checks, not used for training. No held-out benchmark source was accessed. The public datasets were streamed and no full download logs were kept, so those downloads are only partly verified. That does not make this check false.
✓non specialist initializationTraining starts from the general Qwen/Qwen3-1.7B snapshot pinned in architecture.json. The snapshot was downloaded at tr
Training starts from the general Qwen/Qwen3-1.7B snapshot pinned in architecture.json. The snapshot was downloaded at trajectory#48 and passed to evidence/train_drafter.py through --model (trajectory steps 153-219 and 316-345). Round 2 (evidence/qat_finetune.sh, trajectory#451) fine-tunes the dequantised round-1 checkpoint via evidence/packed_to_hf.py, so it descends from the same base. In packed_to_hf.py the Qwen3-8B snapshot is only the tokenizer source. weights/config.json and the weights/model.safetensors inventory show a standard 28-layer, hidden-2048 Qwen3-1.7B layout with tied embeddings and no pretrained speculative or draft heads.
✓no continuation lookupserver.py checks the tokens it receives and calls drafter.draft(tokens, 16). drafter.py does a greedy autoregressive rol
server.py checks the tokens it receives and calls drafter.draft(tokens, 16). drafter.py does a greedy autoregressive rollout with a KV cache over the dequantised Qwen3 weights. The serving path contains no n-gram tables, cached reference continuations, retrieval or network calls. weights/model.safetensors holds only model tensors.
✓actual inference pathserve.sh runs `exec python3 -u server.py`. server.py builds drafter.Drafter from ./weights. drafter.py builds Qwen3 from
serve.sh runs `exec python3 -u server.py`. server.py builds drafter.Drafter from ./weights. drafter.py builds Qwen3 from weights/config.json, dequantises the .q/.s int4 GPTQ tensors in weights/model.safetensors, re-ties lm_head and serves /draft from that model. Trajectory steps 451-455 show the final weights_qat being served and the final architecture.json being written, so the shipped weights are the ones the server loads.
✓harness credentials used only by the harnessThe participant made their own OpenRouter key with `spin openrouter key create` (trajectory#20), tested it at trajectory
The participant made their own OpenRouter key with `spin openrouter key create` (trajectory#20), tested it at trajectory#21, and gen2.sh gets its key from the spin CLI (trajectory#227). evidence/gen_prompts.py reads the participant-supplied OPENROUTER_KEY against the spin proxy. HF_TOKEN was used only for model snapshot downloads (trajectory#46, trajectory#48). Searches for harness setup files, harness credential names or ANTHROPIC keys found nothing.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed3h 12m into the runtook 3m 31s
startup 6.4 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.4 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 486 of about 8,856 held-out requests at concurrency 32: 144.3 s of serving over 31 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 486 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed6h 23m into the runtook 4m 50s
startup 5.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.5 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 485 of about 8,837 held-out requests at concurrency 32: 105.0 s of serving over 26 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 485 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed7h 19m into the runtook 3m 43s
startup 5.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.5 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 486 of about 8,856 held-out requests at concurrency 32: 135.2 s of serving over 27 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 486 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. Final state:
Shipped model — Qwen3-1.7B distilled on Qwen3-8B greedy continuations (non-thinking template, prompt tokens masked), then GPTQ 4-bit group-32 act-order, then a second distillation round: the packed model dequantised to bf16, fine-tuned one epoch (lr 3e-5) on 6,449 fresh continuations from a new prompt pool, and re-quantised. 968.3 MB, 35 files.
Evidence of the improvement (paired per-row agreement, shipped vs. round-1 pack):
- dev2 (320 prompts): +0.0030, t = 4.6
- refs_p2_hold (374 held-out prompts): +0.0020, t = 5.2
- refs_syn_hold (400 unseen-seed programmatic prompts): +0.0011, t = 2.5
All three favour the shipped model; selection used only our own holdouts, never the task's validation split.
Selfcheck (final weights, digest 1aec501a…): passed every gate — startup 5.5 s of 300 s, POST /draft 0.32 s, projected held-out serving 2,464 s of 10,800 s, 0/486 request failures. A real HTTP log against the shipped weights (12 requests, all 200, drafts of 16) is included in evidence/inference_log.jsonl.
Artifact contents: architecture.json (base/target revisions, method, data sources, selection note, 23 training-evidence and 7 inference-evidence files), DATA_MANIFEST.md with the provenance of every corpus (our two LLM-authored pools, the programmatic pool, and public training splits of ultrachat_200k/alpaca/dolly, labels from the named Qwen3-8B revision), download_commands.txt, training logs, packing and evaluation scripts, and data samples.