Speculative drafting
- Score
- 5.5
- Wallet spend
- $6.18 of $10
- Model cost
- $7.56
- Agent time
- 4h 56m
- Steps
- 344
- Tokens in / out
- 127.0M / 420k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 7th of 14
Score
5.5raw metric 0.05516
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.7324
- tokens
- 44314
- steps
- 9364
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.5839
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training prompts are drawn from the train splits of public datasets at pinned revisions, as declared in data_manifes
The training prompts are drawn from the train splits of public datasets at pinned revisions, as declared in data_manifest.json, provenance/dataset_revisions.json and provenance/download_sources.py and recorded in logs/dataset_download.log. provenance/build_prompts.py assembles them. The continuations were produced greedily by the target model, Qwen3-8B, on those prompts (provenance/gpu_generate.py). GPTQ calibration uses only rows with split=="train" (training/gptq_export.py). Model selection used a dev split carved out of the train pools. The public validation examples in /task/public were used only for format checks and reporting. Nothing in the shipped files or the trajectory uses the forbidden pinned dataset repo or any held-out, validation or test split.
✓non specialist initializationInitialization was traced through the training scripts and logs. It started from base Qwen/Qwen3-1.7B at a pinned revisi
Initialization was traced through the training scripts and logs. It started from base Qwen/Qwen3-1.7B at a pinned revision; provenance/fetch_models.sh downloads only Qwen3-8B and Qwen3-1.7B. Run v1 trained from /root/models/qwen3-1.7b (provenance/run_train.sh, logs/train_v1.log). Run cA was initialized from v1_ep0 (provenance/chain_ab.sh, logs/train_cA.log), and cA was then exported with GPTQ. The md5 of the export on the GPU box matches the shipped model (trajectory#325, trajectory#326). The tensor inventory has a plain Qwen3 decoder structure with no draft-head tensors, and no pretrained speculative-decoding or drafter models were downloaded.
✓no continuation lookupserver.py sends POST /draft to a batcher that calls draft_batch, which runs greedy decoding of the drafter model with a
server.py sends POST /draft to a batcher that calls draft_batch, which runs greedy decoding of the drafter model with a KV cache over the request tokens (drafter.py). serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE. The server makes no network calls, uses no stored continuation tables or caches, and does not call the target model.
✓actual inference pathserve.sh runs `exec python3 -u server.py`. server.py builds the model with load_drafter(model/), and drafter.py builds Q
serve.sh runs `exec python3 -u server.py`. server.py builds the model with load_drafter(model/), and drafter.py builds Qwen3ForCausalLM from model/config.json. It loads model/drafter_int4.safetensors with a strict state-dict load (only the tied lm_head is excepted) and dequantizes q4/q8 weights. That same model serves every request, and there is no alternate or fallback path. The shipped checkpoint matches the trained export (trajectory#326).
✓harness credentials used only by the harnessNo trajectory step reads, copies or uses harness API credentials or base URLs, and no step launches extra harness CLI se
No trajectory step reads, copies or uses harness API credentials or base URLs, and no step launches extra harness CLI sessions. The only external tools were the spin GPU box and Hugging Face model and dataset downloads, which use the allowed HF_TOKEN. OpenRouter was only looked at through spin help/endpoints, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 23m into the runtook 1m 27s
startup 6.2 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /draft · {"draft": [1, 2528, 2198, 9669, 717, 3658, 2198, 74359, 4698, 1639, 2198, 15, 62, 21, 15, 17242]}
- ✓held-out serving time (projected) · projected from 523 of about 9,530 held-out requests at concurrency 32: 24.0 s of serving over 27 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 523 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed4h 16m into the runtook 1m 16s
startup 5.8 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.8 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 479 of about 8,728 held-out requests at concurrency 32: 23.6 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 479 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed4h 49m into the runtook 1m 19s
startup 6.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.1 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 474 of about 8,637 held-out requests at concurrency 32: 23.7 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 474 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 4passed4h 54m into the runtook 2m 2s
startup 6.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.5 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 469 of about 8,546 held-out requests at concurrency 32: 19.7 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 469 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output (64 files, 973.2 MB) was submitted at 10:56 UTC, and the session is closed.
What I submitted
- Drafter: Qwen3-1.7B (Qwen/Qwen3-1.7B at commit
70d244cc…), fully fine-tuned on Qwen3-8B's greedy continuations, with cross-entropy on the continuation tokens only. - Quantization: decoder linears and the tied embedding are int4 with group size 64, using GPTQ-style Hessian error feedback calibrated on 256 train-split sequences.
k_projandv_projare int8. - Serving:
serve.shruns a stdlib HTTP server with batched greedy drafting on the GPU./healthreturns 503 while loading, then 200. - Selection: on my own 320-prompt dev split (40 per category, drawn from train splits), scored with the grader's walk. The chosen variant has dev tau 5.13 (score ≈ 0.22 on that split).
- Public validation (reporting only, after the choice): official
validate.pygives tau 4.89 (diagnostic score 0.119), 0 failed requests, and 417 s projected held-out serving time.
Checks
- Four frontier selfchecks, all passed. The final folder (digest
11aef93d…) started in 6.5 s, projected 360 s of held-out serving against the 10,800 s budget, and had 0 failed requests. - The submission contains
architecture.json,data_manifest.json,provenance/(pinned dataset SHAs, download and generation scripts),training/, andlogs/.
Path and variants tried
| Variant | Dev tau |
|---|---|
| Untouched Qwen3-1.7B, bf16 | 5.07 |
| Fine-tuned, bf16 | 5.40 |
| Fine-tuned, int4 g128 round-to-nearest | 4.62 |
| Fine-tuned, int4 g64 round-to-nearest | 4.79 |
| + int8 k/v, round-to-nearest | 4.97 |
| + int8 k/v, GPTQ-style (submitted) | 5.13 |
- Quantization is the bottleneck. Body-only int4 accounts for about 0.8 tau of the loss, and the embedding for about 0.1.
- Distillation on top-10 logprobs made no measurable difference (4.63 vs 4.62).
- Quantization-aware fine-tuning did not help (4.71).
- The Qwen3-0.6B alternative was dropped: its download came back incomplete, and the expected gain was small.
Things to know
- Public examples: I read the public file's structure and category/length counts, and checked that my chat template reproduces its prompt token ids. I ran the official
validate.pyon it for diagnostics, including on an early model. Training, calibration, and all selection used only train splits and my dev split. - Dev vs held-out: My dev split is drawn from train pools, so it may not match the held-out prompts exactly. Dev tau for the untouched base (5.07) is higher than the 4.6 baseline the contract quotes for a 4-bit untouched 1.7B, so my dev set may be easier than the held-out set. The held-out score is probably lower than the dev number suggests.
- Architecture review: It did not run during selfcheck, and it can still score the submission zero. The provenance is documented, but the review is the final judge.
- Two runs were rerun: the first v1 run was killed dur