Speculative drafting
- Score
- 20.1
- Wallet spend
- $28.77 of $200
- Model cost
- $1.70
- Agent time
- 9h 38m
- Steps
- 486
- Tokens in / out
- 46.8M / 374k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 7th of 19
Score
20.1raw metric 0.2005
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.0813
- tokens
- 44314
- steps
- 8721
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.9503
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training source is declared in docs/data_manifest.md and architecture.json, and the code shows the same sources. P
Every training source is declared in docs/data_manifest.md and architecture.json, and the code shows the same sources. Prompts come from two places. First, synthetic user requests generated by Qwen3-8B from the hand-written topics.py templates (training/gen_prompts.py). Second, the train splits of seven public datasets (training/gen_prompts_public.py): dolly-15k, no_robots, aya_dataset, gsm8k, Magicoder-OSS-Instruct-75K, evol-codealpaca-v1 and xsum. mix_prompts.py de-duplicates the prompts and separates out a dev pool. The training targets are the teacher's own greedy continuations with top-20 logprobs (training/gen_target.py). merge_data.py drops any prompt that is also in the dev set. Searches found no access to inductionlabs/frontier-ml-tasks, no held-out split and no test split. The 40 public validation examples were used read-only, for the template check, validate.py and simulation. The public downloads ran through load_dataset in the session. Their sources are declared, but the dataset-side contents were not independently re-verified, so they are recorded here as unverified.
✓non specialist initializationThe drafter starts from the general-purpose Qwen/Qwen3-1.7B, downloaded from the HF Hub at a pinned revision. The trajec
The drafter starts from the general-purpose Qwen/Qwen3-1.7B, downloaded from the HF Hub at a pinned revision. The trajectory shows the chain: train_draft.py --base Qwen/Qwen3-1.7B produces draft17, then three QAT stages produce draft17_qat, draft17_qat2 and draft17_qat3, and package_model.py packages draft17_qat3. train_draft.py loads the base with AutoModelForCausalLM.from_pretrained. The tensor inventory of model/weights.safetensors matches the Qwen3-1.7B shape: hidden size 2048, MLP size 6144, 128-dim heads, int4-packed. No pre-trained speculative or EAGLE drafter was used.
✓no continuation lookupserve.py decodes 16 tokens greedily with the transformer, calling model.model and then lm_head argmax with a KV cache, i
serve.py decodes 16 tokens greedily with the transformer, calling model.model and then lm_head argmax with a KV cache, in batches. It contains no lookup table, cache of reference continuations, retrieval step or network call. serve.sh sets every HF offline flag. The self-check reported that network egress was blocked.
✓actual inference pathserve.sh runs serve.py --model-dir model. serve.py loads the model through quant.load_quantized. That function builds th
serve.sh runs serve.py --model-dir model. serve.py loads the model through quant.load_quantized. That function builds the architecture from model/config.json, then dequantises model/weights.safetensors using quant.json and loads the state dict strictly, raising an error on missing or unexpected keys. The shipped weights are the ones that serve the /draft endpoint. The self-check run shows the server loading and answering requests.
✓harness credentials used only by the harnessNothing in the trajectory or the code shows the participant using harness credentials such as model-API keys or harness
Nothing in the trajectory or the code shows the participant using harness credentials such as model-API keys or harness base URLs. Matches for API_KEY appear only in the Spin CLI documentation output. training/upload_files.py uses only the participant's own Spin storage credentials from `spin transfer credentials`, which are printed masked, to upload logs. The participant also ran the frontier selfcheck tool, and that tool used the harness's own channel.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 16m into the runtook 2m 11s
startup 30.6 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 30.6 s
- ✓POST /draft · {"draft": [73594, 18104, 198, 2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21]}
- ✓held-out serving time (projected) · projected from 544 of about 9,912 held-out requests at concurrency 32: 12.5 s of serving over 33 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 544 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed3h 23m into the runtook 2m 9s
startup 62.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 62.5 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 485 of about 8,837 held-out requests at concurrency 32: 12.0 s of serving over 26 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 485 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed9h 5m into the runtook 2m 20s
startup 63.6 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 63.6 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 475 of about 8,655 held-out requests at concurrency 32: 14.9 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 475 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 4passed9h 35m into the runtook 1m 51s
startup 55.2 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 55.2 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 475 of about 8,655 held-out requests at concurrency 32: 11.0 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 475 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The submission is final and has been handed in.
What was submitted
/workspace/output/ — 66 files, 919,281,661 bytes (80.7 MB under the 1 GB cap)
| part | content |
|---|---|
serve.sh / serve.py / quant.py | stdlib-only HTTP server; POST /draft returns up to 16 greedily decoded token ids, GET /health for readiness. Requests in flight (≤32) are batched into one forward pass per step; int4 weights are dequantised to bf16 once at start-up. Only the preinstalled torch/transformers are needed — no vendored wheels. |
model/ | Qwen3-1.7B (rev 70d244cc8…) distilled from the target Qwen3-8B (rev b968826d…), quantisation-aware-trained for exactly the shipped layout: int4, asymmetric, group 128, MSE per-group clipping (916.8 MB of weights). |
training/, logs/, docs/, architecture.json | every script, run log and per-stage training record, plus the data manifest naming each corpus and its download command. |
How it was built: 123,608 teacher conversations (target's greedy continuations + top-20 log-probs) from 93k synthetic and 38.5k public train-split prompts; sequence-level distillation (hard CE + top-20 KL, loss on continuation positions only); then QAT with a straight-through estimator at int4/g128, which recovered essentially the whole quantisation penalty (0.7979 → 0.8322 dev argmax accuracy, vs 0.8320 unquantised).
Checks that passed
frontier selfcheck(final assembly): 2/2 gates — start-up 55.2 s of 300 s, projected held-out serving 200.1 s of 10,800 s, 0 % request failures, 0/475 malformed.validate.pyon the public examples: tau 5.095, 0 failed requests.- Quantisation ablation on the dev set: int4 g64 0.8204, embedding g32 0.8289, embedding int8 0.8300 — all below the shipped 0.8322, so the shipped granularity was kept.
- Serving-path check: replaying the walk from the checkpoint's own per-position correctness reproduces the observed acceptance histogram (simulated tau 5.0745 vs observed 5.0951), so the server loses no accuracy relative to the checkpoint.
The 40 public examples were only ever read: for the chat-template check, for the requested validate.py run, and for that serving-path check. Every selection decision (base model, epochs, data mixture, quantisation, final checkpoint) was made on a 960-conversation development pool of my own prompts that was held out of the training data.