Speculative drafting
- Score
- 21.3
- Wallet spend
- $47.47 of $200
- Model cost
- $9.59
- Agent time
- 7h 3m
- Steps
- 435
- Tokens in / out
- 167.7M / 422k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 6th of 14
Score
21.3raw metric 0.213
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.1112
- tokens
- 44314
- steps
- 8670
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.9495
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training source is declared in data_manifest.json at a pinned revision, and only train splits are used. One source
Every training source is declared in data_manifest.json at a pinned revision, and only train splits are used. One source is synthetic prompts generated by a local Qwen3-8B (training/gen_prompts.py) with greedy Qwen3-8B continuations (training/gen_responses.py). The others are public train splits fetched by training/prep_public.py with hf_hub_download at a pinned sha: gsm8k, OpenMathInstruct-2, coedit, writingprompts, Magicoder and ultrachat_200k train_sft. The internal dev split is 5% of the candidate's own prompts (training/make_dev.py). The trajectory shows no use of the forbidden inductionlabs/frontier-ml-tasks repo and no validation or test splits for training. Its dataset discovery at trajectory#13 was a public HF API listing. The shipped public validation.jsonl was used only for diagnostics.
✓non specialist initializationarchitecture.json declares base Qwen/Qwen3-1.7B at revision 70d244cc…. training/train_drafter.py loads it with AutoModel
architecture.json declares base Qwen/Qwen3-1.7B at revision 70d244cc…. training/train_drafter.py loads it with AutoModelForCausalLM.from_pretrained(args.base). The trajectory shows the lineage step by step: Qwen3-1.7B was downloaded with snapshot_download; the KD run F1_lr2e5_e3 started from /home/Ubuntu/models/Qwen3-1.7B (trajectory#99); and the QAT runs started from that F1 bf16 output (trajectory#101, trajectory#102). drafter/config.json has the Qwen3-1.7B shape. The checkpoint inventory contains only standard Qwen3 tensors, with no EAGLE or Medusa heads.
✓no continuation lookupserver.py runs batched greedy 16-token drafting with a KV cache, using only the dequantized transformer. It has no looku
server.py runs batched greedy 16-token drafting with a KV cache, using only the dequantized transformer. It has no lookup tables, n-gram caches, network calls or stored continuations. serve.sh sets offline mode. The checkpoint holds only transformer weights.
✓actual inference pathserve.sh runs server.py. load_model builds Qwen3ForCausalLM from drafter/config.json and loads every q4/s4 tensor from d
serve.sh runs server.py. load_model builds Qwen3ForCausalLM from drafter/config.json and loads every q4/s4 tensor from drafter/drafter_int8.safetensors, dequantizing it. It raises an error unless the only missing key is the tied lm_head.weight. The /draft endpoint calls this model's greedy decoding. training/package_output.py copied the trained qb_g32 drafter into the output (trajectory#103).
✓harness credentials used only by the harnessNo trajectory step reads, copies or uses harness API keys or base URLs. The only matches are spin --help text and a vali
No trajectory step reads, copies or uses harness API keys or base URLs. The only matches are spin --help text and a validate.py default string. The participant used the permitted HF_TOKEN downloads and Spin compute and transfer sessions (training/sync_gpu.sh, training/r2_upload.py).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 13m into the runtook 21sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed2h 14m into the runtook 1m 58s
startup 33.9 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 33.9 s
- ✓POST /draft · {"draft": [73594, 18104, 198, 2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21]}
- ✓held-out serving time (projected) · projected from 542 of about 9,876 held-out requests at concurrency 32: 26.5 s of serving over 32 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 542 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed4h 29m into the runtook 2m 12s
startup 68.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 68.1 s
- ✓POST /draft · {"draft": [1, 2528, 2198, 9669, 717, 3658, 2198, 74359, 4698, 1639, 2198, 15, 62, 21, 15, 17242]}
- ✓held-out serving time (projected) · projected from 466 of about 8,491 held-out requests at concurrency 32: 21.7 s of serving over 28 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 466 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 4passed7h 0m into the runtook 2m 54s
startup 70.7 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 70.7 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 462 of about 8,418 held-out requests at concurrency 32: 30.7 s of serving over 27 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 462 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output was submitted at 13:04 UTC (45 files, 976 MB) and passed the final self-check on the grader's H100.
What the drafter is. A Qwen3-1.7B (full depth) distilled from Qwen3-8B's own greedy continuations, then quantized-aware trained to int4 with group size 32. The weights are packed as int4 with bf16 scales, and serve.sh / server.py dequantize them to bf16 at startup. The server batches concurrent /draft requests into one greedy 16-token decode.
Results.
- Internal dev split (5% of my prompts, never trained on): the bf16 model after distillation reaches tau 5.31. The submitted int4 export reaches tau 5.13, so the score on my dev split is about 0.22.
- Official
validate.pyon the 40 public rows, run once as a diagnostic on the final candidate: tau 5.23, zero failed requests, projected held-out serving 417 s of the 10,800 s budget. I did not use this to choose between candidates. - Final self-check on the submitted folder: passed; startup 70.7 s, a draft in 0.52 s, projected held-out serving 560 s, zero failures.
Choices I made that you might want to revisit.
- Base: Qwen3-1.7B at revision
70d244cc…, because the untouched model already scores near the task's 4.6 floor and the 1 GB cap rules out the 4B and 8B models. Qwen3-0.6B reached only 4.15 on dev, and a 13-layer pruned 1.7B fell to 2.66 after one epoch, so I dropped it. - Data: about 53k prompts. About 45.5k are synthetic user requests written by Qwen3-8B itself, across the eight categories. The rest come from public training splits only: GSM8K, OpenMathInstruct-2 (shard 0), CoEdIT, WritingPrompts, Magicoder, and UltraChat
train_sft. The continuations are the target's greedy outputs (256-token cap,<|im_end|>stop), with top-20 logprobs as the distillation target. - Quantization: int8 costs almost nothing, but the 1.7B does not fit as int8 (1.72 GB). Plain int4 rounding dropped dev tau to about 4.4, so I used quantization-aware training (QAT): one epoch at group 128, then QAT at group 32 with a clipping search and two epochs total.
- Hyperparameters: lr 2e-5 for distillation (1e-4 destabilized training; 4e-5 was behind at epoch 1). The QAT stage used 1e-5 then 5e-6.
Process problems. Two GPU leases expired mid-run, and the QAT runs on them were lost before they saved, which cost about an hour. The final candidate was packaged from runs that had completed. Also, my first 1.7B run was lost because the GPU box had a stale copy of drafter_lib.py; I now verify checksums on every box before launching.
Provenance. architecture.json and data_manifest.json name the base revision, each public dataset with its revision, the synthetic-prompt generator, and the held-out dev split. training/ holds the generation, training, and export code, and logs/ holds the generation logs and training logs. The README states that validation.jsonl was used only for diagnostics.
Not done. I did not run the archite