Speculative drafting
- Score
- 16.2
- Wallet spend
- $23.02 of $50
- Model cost
- $1.71
- Agent time
- 7h 43m
- Steps
- 678
- Tokens in / out
- 67.3M / 574k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 6th of 19
Score
16.2raw metric 0.1622
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.9892
- tokens
- 44314
- steps
- 8882
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.8554
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training targets are greedy continuations from Qwen3-8B, generated by training/gen_target.py with vLLM at a pinned r
The training targets are greedy continuations from Qwen3-8B, generated by training/gen_target.py with vLLM at a pinned revision, temperature 0. The prompts come from three declared sources. (1) Public HF train splits loaded through load_hf(split="train") in prep_prompts.py and prep_prompts2.py, including gsm8k, aqua_rat, Magicoder, self-oss-instruct, nq_open, trivia_qa, dolly, ultrachat, no_robots, smoltalk, MATH and aya; data_manifest.json lists them with revisions. (2) Seeded programmatic generators. (3) Prompts written by the teacher model. The final stage (run_round7.sh) adds only the HuggingFaceH4/no_robots train split through fetch_human_prompts.py. The selection splits (dev2, dev4, dev6) are self-generated holdouts. The shipped /task/public/validation.jsonl was copied into task_copy and used only for the shingle-overlap filter (filter_public.py) and for reporting evaluation, which is allowed. No search hit shows a test split or a pinned/forbidden corpus being loaded. Most sources have download commands or logs (run_all.log); any source without a retained log is a declared public source and is recorded here as unverified.
✓non specialist initializationtrain_drafter.py initializes from AutoModelForCausalLM.from_pretrained(Qwen/Qwen3-1.7B). Its only resume option loads th
train_drafter.py initializes from AutoModelForCausalLM.from_pretrained(Qwen/Qwen3-1.7B). Its only resume option loads the participant's own .pt checkpoints from earlier rounds (run_round7.sh: --base Qwen/Qwen3-1.7B --resume ckpt/r17q/...). The download scripts and logs call snapshot_download only for Qwen/Qwen3-1.7B. model/config.json matches Qwen3-1.7B (hidden 2048, 28 layers, intermediate 6144, tied embeddings). The model.safetensors inventory holds only standard Qwen3 tensors for layers 0-27, plus norm, embed_tokens and an nf4 codebook. There are no EAGLE, Medusa or draft-head tensors, and no pre-trained speculative drafter was obtained.
✓no continuation lookupThe /draft handler in server.py truncates the request to its last 4096 tokens and queues it. A batch worker then calls d
The /draft handler in server.py truncates the request to its last 4096 tokens and queues it. A batch worker then calls drafter.generate_drafts, which does greedy KV-cached decoding with the local model for 16 new tokens. There are no lookup tables, no caches of target outputs, no network calls and no target model at serve time. serve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1.
✓actual inference pathThe serving path is serve.sh -> python server.py --model-dir $HERE/model -> drafter.load_model. load_model builds the mo
The serving path is serve.sh -> python server.py --model-dir $HERE/model -> drafter.load_model. load_model builds the model from model/config.json, loads model/model.safetensors and dequantizes the NF4 qweight/scales. The load is strict and raises on any key mismatch except the tied lm_head. Drafts then come from generate_drafts with that model. model/model.safetensors is the only checkpoint shipped, and no alternate serving path exists.
✓harness credentials used only by the harnessThe trajectory shows no use, printing or copying of harness API keys or base URLs, and no extra harness sessions launche
The trajectory shows no use, printing or copying of harness API keys or base URLs, and no extra harness sessions launched as workers. Searches for ANTHROPIC, printenv, claude -p and openrouter found matches only in the Spin CLI help/policy text. Remote compute ran through the spin CLI. The only environment probe was `env | grep -i -c HF_TOKEN`, which counts matching lines without revealing any value.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 10m into the runtook 3m 1s
startup 26.2 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 26.2 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 554 of about 10,095 held-out requests at concurrency 32: 62.5 s of serving over 33 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 554 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed4h 49m into the runtook 2m 57s
startup 55.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 55.5 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 466 of about 8,491 held-out requests at concurrency 32: 40.9 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.2 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 466 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed6h 4m into the runtook 3m 7s
startup 67.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 67.1 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 444 of about 8,090 held-out requests at concurrency 32: 58.3 s of serving over 32 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 444 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 4passed7h 38m into the runtook 2m 52s
startup 62.0 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 62.0 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 447 of about 8,145 held-out requests at concurrency 32: 53.0 s of serving over 33 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 447 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Submitted (2026‑10‑21 03:19:33 UTC). Final state of /workspace/output — 73 files, 971,556,948 bytes (limit 1,000,000,000):
Drafter — Qwen3‑1.7B (rev 70d244cc8…), weights stored as 4‑bit NF4 (group 32, per‑group fp16 absmax scales) and dequantized to bf16 at load → 968.1 MB of weights, no quantization runtime needed. serve.sh starts in 62 s and POST /draft returns up to 16 token IDs in ~0.29 s.
Training — sequence‑level distillation from Qwen3‑8B's own greedy, non‑thinking continuations (chat template reproduced the eval's prompt_tokens 40/40). The last two stages were quantization‑aware: the exact deployed NF4 rounding is simulated in the forward pass with a straight‑through estimator, and test_fake_quant.py shows the simulated weights are bit‑identical to what the export writes and the server reconstructs. The final stage added 8,178 human‑written prompts (no_robots train split only) answered by the target, because the evaluation conversations are human‑written.
Selection (held‑out dev splits of my own corpus; no public validation/test split was used for training, calibration or selection):
| model | dev4 τ | dev6 τ (human) | public‑40 τ |
|---|---|---|---|
| previous (no QAT) | 5.6449 | – | 5.0108 |
| QAT stage 1 | 5.9611 | 4.4782 | 5.2411 |
| QAT stage 2 (shipped) | 5.9703 | 4.5931 | 5.3140 |
A candidate had to win on both dev splits; only one of the two stage‑2 checkpoints passed. The public‑40 was only used to confirm the server reproduces the offline measurement, never to pick a checkpoint.
Checks — 4/4 self‑checks used; the final one passed every gate: size, startup 62 s (limit 300), draft latency 0.288 s, projected held‑out serving 966 s (limit 10,800), 0 % request failures, healthy after requests. The serving path was also verified end‑to‑end offline against validate.py.