Speculative drafting
- Score
- 37.0
- Wallet spend
- $39.11 of $50
- Model cost
- $64.38
- Agent time
- 7h 10m
- Steps
- 223
- Tokens in / out
- 51.2M / 225k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 1st of 14
Score
37.0raw metric 0.3699
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.4878
- tokens
- 44314
- steps
- 8075
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 6.3379
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.json names three training sources: the training splits of public datasets (collected by training/collect_p
data_manifest.json names three training sources: the training splits of public datasets (collected by training/collect_public.py with split='train' or 'train_sft'; the log is retained in training/logs/collect_public.txt), synthetic prompts from training/gen_prompts.py (category, domain and tone specs sent through spin openrouter), and continuations generated by the target Qwen3-8B. Checkpoints are selected on dev_all.jsonl, a hash-held-out ~1% of the generated training rows (chain_f.sh --select_set, train3.py). Public validation data was used only for reporting. Every source is declared, and nothing indicates that evaluation data was used for training or selection.
✓non specialist initializationarchitecture.json declares Qwen/Qwen3-1.7B at revision 70d244cc… as the base. training/chain_f.sh runs train3.py with --
architecture.json declares Qwen/Qwen3-1.7B at revision 70d244cc… as the base. training/chain_f.sh runs train3.py with --model Qwen/Qwen3-1.7B at that revision and no --init, so run F starts from the general base model rather than an existing drafter or speculative-decoding checkpoint. model/weights.json shows standard Qwen3-1.7B tensor names and shapes. The trajectory shows no download of a pretrained drafter or EAGLE checkpoint.
✓no continuation lookupserver.py drafts greedily with argmax from the neural model for DRAFT_LEN=16 tokens, using a KV cache. It contains no n-
server.py drafts greedily with argmax from the neural model for DRAFT_LEN=16 tokens, using a KV cache. It contains no n-gram table, cached continuation store, retrieval or network call, and serve.sh forces HF offline mode. wcodec.py only quantizes, dequantizes and does lzma I/O. The weight inventory contains only transformer tensors.
✓actual inference pathserve.sh execs server.py. server.py calls wcodec.load on model/weights, builds AutoModelForCausalLM from model/config.js
serve.sh execs server.py. server.py calls wcodec.load on model/weights, builds AutoModelForCausalLM from model/config.json on the meta device, and loads the state dict with strict=True and assign=True. That model then produces the served drafts. The custom wcodec format explains why the generic inspector could not read model/weights.bin, and this is not a violation. In trajectory#215, the trained final_F weights are copied into the output and verified with cmp.
✓harness credentials used only by the harnessThe task prompt (trajectory#1) gives the participant HF_TOKEN as a read-only token for downloading models and datasets.
The task prompt (trajectory#1) gives the participant HF_TOKEN as a read-only token for downloading models and datasets. The participant used it for exactly that, exporting it on the leased GPU box. The OpenRouter key came from the participant's own spin openrouter tooling (trajectory#17), and gen_prompts.py reads it from /tmp/or_key.json. No harness-only or grader credential was accessed or used. The participant scanned the output for secrets before finishing (trajectory#222), and no secrets are shipped.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 31m into the runtook 3m 49s
startup 17.7 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 17.7 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 465 of about 8,473 held-out requests at concurrency 32: 34.1 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 465 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed4h 11m into the runtook 1m 39s
startup 17.6 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 17.6 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 435 of about 7,926 held-out requests at concurrency 32: 30.3 s of serving over 26 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 435 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed7h 8m into the runtook 1m 18s
startup 15.7 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 15.7 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 446 of about 8,127 held-out requests at concurrency 32: 22.4 s of serving over 26 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 446 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
I've submitted the drafter. It reaches tau 5.52 on the public examples when measured with validate.py against the real server, which would score about (5.52 − 4.6) / 2.4 ≈ 0.38. On my own held-out split of generated data (synthetic prompts only, the closest match to the hidden set) it reaches 5.55. The final self-check passed: startup took 15.7 s (limit 300 s), projected held-out serving time is 408 s (limit 10,800 s), no requests failed, and the submission is 967.3 MB.
What it is
- Qwen3-1.7B (revision
70d244cc…), fully fine-tuned on Qwen3-8B's greedy, non-thinking continuations of 212k prompts. 177k were synthetic prompts in the eight categories, generated throughspin openrouterwithout showing the generator any evaluation prompts. The other 35k came from public training splits (Tulu-3, Dolly, GSM8K train, Magicoder, UltraChat, Aya). - The loss combined the target's greedy tokens with matching its full next-token distribution, using Qwen3-8B as a live teacher.
- To fit in 1 GB, the weights are stored at about 4.5 bits each and expanded back to full precision on the GPU at startup (about 9 s). Training simulated that rounding from the first step, so the stored model scores the same as the trained one.
- The server uses only the preinstalled torch and transformers and always returns 16 draft tokens.
What I tried and dropped
- Qwen3-4B: scores 6.4 unquantized, but squeezed to about 2 bits per weight to fit 1 GB it fell to 3.3, so I stayed with 1.7B.
- Several shorter training phases: these plateaued at about 5.41. One long run over all the data did clearly better (5.55 vs 5.36 on the synthetic split).
- One crashed run: the GPU box's disk filled up mid-run. I lost that run and freed space afterwards; it didn't affect the final model.
The public examples were used only to report scores, never for training, calibration or picking checkpoints. Provenance is in architecture.json and data_manifest.json, with scripts and logs under training/.
Total spend was $39.10 of the $50 budget, and all GPU machines were released before submitting.