Speculative drafting
- Score
- 15.6
- Wallet spend
- $9.90 of $10
- Model cost
- $105.27
- Agent time
- 5h 59m
- Steps
- 108
- Tokens in / out
- 22.1M / 211k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 4th of 14
Score
15.6raw metric 0.1559
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.9741
- tokens
- 44314
- steps
- 8909
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.8379
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTraining prompts come from public sources. data_prep/collect_prompts.py downloads them with hf_hub_download: the train s
Training prompts come from public sources. data_prep/collect_prompts.py downloads them with hf_hub_download: the train splits of gsm8k, hendrycks_math, EditPackFT, smoltalk, aya and Magpie, plus the unsplit public corpora Magicoder-OSS, CodeFeedback, dolly-15k and the oasst2 full prompt dump. It keeps only the prompts. More prompts are generated by Qwen3-8B (data_prep/gen_synth_prompts.py). The training targets are greedy continuations from Qwen3-8B itself (data_prep/gen_responses.py). All of these sources are declared in DATA_MANIFEST.md and the downloads are logged. The public /task/public validation.jsonl was used only for fidelity checks and evaluation. Nothing in the code, the logs or the trajectory touches the pinned benchmark dataset repo or any validation/test split.
✓non specialist initializationtraining/train.py calls from_pretrained(args.init_from or args.base). In training/train17_v2.sh and logs/train17_v2.log,
training/train.py calls from_pretrained(args.init_from or args.base). In training/train17_v2.sh and logs/train17_v2.log, init_from is null and base is the Qwen/Qwen3-1.7B snapshot at a pinned revision, downloaded by data_prep/gpu_setup.sh. That is a general-purpose model, not a speculative drafter. The shipped checkpoint holds only the standard Qwen3 tensors (embed, 28 layers, norm) in int4 form. It has no EAGLE/Medusa-style heads and no head_* tensors.
✓no continuation lookupserver.py only filters and truncates the request tokens, then calls ENGINE.draft. qwen3_engine.py runs greedy autoregres
server.py only filters and truncates the request tokens, then calls ENGINE.draft. qwen3_engine.py runs greedy autoregressive decoding with a KV cache over the loaded weights. Neither file has a network call, a cache of benchmark continuations, a lookup table or a dataset file read.
✓actual inference pathserve.sh runs server.py with --weights ./weights. server.py builds Qwen3Draft from that directory, and qwen3_engine.py l
serve.sh runs server.py with --weights ./weights. server.py builds Qwen3Draft from that directory, and qwen3_engine.py loads weights/config.json and weights/model.safetensors, dequantizes the int4 groups and decodes with them. In trajectory#106 the md5 of the shipped model.safetensors matches the train17_v2 export (engines/q17_v2_int4), so the served weights are the trained model. The experimental correction head in logs/fit_head.log was not shipped, and the weight inventory has no head_* tensors.
✓harness credentials used only by the harnessThe participant used the spin CLI as documented: compute exec, s5cmd, and wallet-scoped transfer credentials to move its
The participant used the spin CLI as documented: compute exec, s5cmd, and wallet-scoped transfer credentials to move its own data to and from its GPU box. The participant's code makes no model-API calls with harness credentials and starts no extra agent sessions. HF_TOKEN was used only for model and dataset downloads. No credentials ship in the submission: there is no session.json and serving needs no network.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 27m into the runtook 1m 41s
startup 6.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.5 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 580 of about 10,568 held-out requests at concurrency 32: 24.9 s of serving over 31 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 580 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed4h 50m into the runtook 1m 14s
startup 5.9 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.9 s
- ✓POST /draft · {"draft": [8420, 374, 279, 821, 304, 27445, 3561, 448, 279, 7102, 1172, 3055, 1447, 13874, 3989, 2528]}
- ✓held-out serving time (projected) · projected from 470 of about 8,564 held-out requests at concurrency 32: 15.3 s of serving over 28 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 470 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Only the intended processes remain (the main 1.7B run and the new chain). Next milestone is the step-2000 eval around 02:48 UTC; polling then.