Speculative drafting
- Score
- 8.7
- Wallet spend
- $34.52 of $200
- Model cost
- $13.15
- Agent time
- 6h 49m
- Steps
- 270
- Tokens in / out
- 59.7M / 292k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 8th of 14
Score
8.7raw metric 0.08682
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.8084
- tokens
- 44314
- steps
- 9216
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.6350
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training, calibration and selection data all have declared sources. train/collect_prompts.py, train/DATA.md and trai
The training, calibration and selection data all have declared sources. train/collect_prompts.py, train/DATA.md and train/prompts_manifest.json name public dataset train splits (tulu, smoltalk, gsm8k, trivia, aya, fineweb, codealpaca) plus synthetic templates. The responses are greedy Qwen3-8B continuations. The GPTQ calibration rows in train/gptq.py and train/gptq.log come from the same training data. Selection and evaluation in train/eval_tau.py use only the public /task/public validation examples. Trajectory steps 66-67 show distill.jsonl and train.jsonl were built only from these declared sources. No test split or held-out data was used. The downloads themselves were only partly retained, beyond setup and collect log summaries, so the public sources are recorded as unverified-by-log, but they are declared.
✓non specialist initializationarchitecture.json declares the base as Qwen/Qwen3-1.7B at a pinned revision. The trajectory shows the chain: snapshot_do
architecture.json declares the base as Qwen/Qwen3-1.7B at a pinned revision. The trajectory shows the chain: snapshot_download of Qwen3-1.7B, then drafter_17, drafter_final (KD), GPTQ w4g2_final, then the LoRA recovery adapters. The checkpoint structure matches Qwen3-1.7B: hidden size 2048, intermediate size 6144, k/v proj 1024, packed int4 with scale, zero and perm. The LoRA has rank 16, with 196 adapters (28 layers x 7). No EAGLE or other pretrained speculative-decoding drafter was downloaded or used.
✓no continuation lookupIn server.py, the default mode is greedy autoregressive drafting by the loaded model. The optional lookup, switch and in
In server.py, the default mode is greedy autoregressive drafting by the loaded model. The optional lookup, switch and interleave modes only call longest_suffix_match over the current request's own tokens. There are no stored continuation tables, no corpus or datastore lookups and no network calls. serve.sh sets offline mode.
✓actual inference pathserve.sh runs server.py with MODEL_DIR=weights. server.py calls loader.load_drafter, which reads weights/model_int4.safe
serve.sh runs server.py with MODEL_DIR=weights. server.py calls loader.load_drafter, which reads weights/model_int4.safetensors and weights/.meta.json, dequantizes the int4 tensors, and merges weights/lora.safetensors with lora.json. It then builds the model from config.json and asserts no missing or unexpected keys. All drafts come from this model. Trajectory step 255 shows the server log reporting 'merged 196 LoRA adapters' and 'mode=model'. Trajectory step 259 shows the shipped weights were copied from the same w4g2_final directory.
✓harness credentials used only by the harnessThe only credential handling in the trajectory was HF_TOKEN, used for Hugging Face model and dataset downloads on the sp
The only credential handling in the trajectory was HF_TOKEN, used for Hugging Face model and dataset downloads on the spin compute box, which the policy allows. Searches found no harness API keys, base URLs, OpenRouter, Anthropic or Claude/codex CLI launches, and no copying of harness credentials into the submission code. The shipped serve.sh and server.py run fully offline.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed6h 43m into the runtook 4m 33s
startup 75.6 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 75.6 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 464 of about 8,455 held-out requests at concurrency 32: 130.2 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 464 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final summary:
Result: distilled speculative drafter for Qwen3-8B (non-thinking, greedy) achieving tau 4.95 on the 40 public conversations (score ≈ 0.14), in a 967MB submission that passed all selfcheck gates.
Approach
- Base: Qwen3-1.7B (not a drafting model), distilled on 202k prompts / 36M tokens of target greedy continuations generated with vLLM on a rented H100. Prompt sources were all training splits or pretraining corpora (Tulu-3, GSM8K-train, TriviaQA-train, Aya-train non-English, FineWeb-Edu-train, CodeAlpaca-train) plus original synthetic templates in the 8 eval categories; plus on-policy (drafter-state) augmentation.
- Training: SFT → on-the-fly full-vocab KL + hard-position upweighting (fp tau 5.18, up from 5.07 base; 0.6B capped at 4.29).
- Fit-to-budget: custom GPTQ group-128 int4 (914MB) + rank-16 LoRA recovery adapters (35MB, trained with KD), both dequantized/merged to bf16 once at server startup — zero inference overhead. Naive RTN int4 collapsed to 4.03; GPTQ gave 4.49; LoRA recovered to 4.95.
- Serving: plain greedy 16-token drafts, stdlib HTTP server, requests serialized on a lock. Prompt-lookup hybrids and beam decoding were tested offline and rejected (hurt or broke the serving budget).
Verification: selfcheck passed everything — size, startup 75.6s/300s, 0.30s/request, projected held-out serving 2373s/10800s, 0% failures. Provenance (scripts, manifests, trimmed logs) ships under train/ with the base revision recorded in architecture.json.