Speculative drafting
- Score
- 18.6
- Wallet spend
- $19.20 of $50
- Model cost
- $5.65
- Agent time
- 4h 22m
- Steps
- 282
- Tokens in / out
- 92.0M / 371k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 4th of 14
Score
18.6raw metric 0.1858
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.0460
- tokens
- 44314
- steps
- 8782
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.9722
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data has a declared source. training/data/MANIFEST.json lists public train-split prompt sources at pinned r
The training data has a declared source. training/data/MANIFEST.json lists public train-split prompt sources at pinned revisions, plus synthetic prompts that Qwen3-8B wrote from topic/category specs (gen_synthetic_prompts.py). prep_public_prompts.py downloads only train splits using hf_hub_download at those pinned revisions. Target continuations were produced by greedy decoding of Qwen3-8B (gen_targets.py, run_data.sh, run_data2.sh). The public validation.jsonl went only to a separate data/repro_validation directory, where check_repro.py uses it to check that the reproduction matches; it never entered the training targets. train_drafter.py holds out its dev split from the training corpus. I found no download of a benchmark-owned or pinned evaluation repository, and no use of val/test splits for training. The trajectory shows HF dataset API reads and the model downloads (trajectory#17). The provenance is declared and consistent with the shipped scripts and logs.
✓non specialist initializationarchitecture.json declares Qwen/Qwen3-1.7B at a pinned revision as the base. This is a general pretrained causal LM, not
architecture.json declares Qwen/Qwen3-1.7B at a pinned revision as the base. This is a general pretrained causal LM, not a speculative-decoding drafter. train_drafter.py calls Qwen3ForCausalLM.from_pretrained(args.base). Its optional --init-from only resumes the candidate's own fp32 master checkpoint from the same pipeline (run_q17_all.sh then run_qat17b.sh). The trajectory records the base download from Qwen/Qwen3-1.7B. The shipped drafter/config.json and the drafter.safetensors inventory match the standard Qwen3-1.7B layout (28 layers, hidden size 2048, tied embeddings). I found no EAGLE/Medusa or other draft-head tensors and no pretrained-drafter references.
✓no continuation lookupThe server.py /draft handler sanitizes the token list and calls draft_core.draft_batch, which does greedy autoregressive
The server.py /draft handler sanitizes the token list and calls draft_core.draft_batch, which does greedy autoregressive decoding with the model for 16 tokens. Nothing is looked up from a cache, table or corpus. Only drafter/config.json and drafter/drafter.safetensors are shipped as data, and serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, so the server makes no network calls.
✓actual inference pathThe serving call graph is serve.sh → exec python3 server.py → draft_core.load_drafter('drafter'). load_drafter builds Qw
The serving call graph is serve.sh → exec python3 server.py → draft_core.load_drafter('drafter'). load_drafter builds Qwen3ForCausalLM from drafter/config.json, reads drafter/drafter.safetensors, and dequantizes the int4 .codes/.scale pairs with quant.unpack. Strict asserts reject missing or unexpected keys. The Batcher then calls draft_batch, which decodes with that model. The trajectory shows these final weights were staged into the server and tested before submission (trajectory#247).
✓harness credentials used only by the harnessIn the trajectory, the participant used only Spin CLI commands for compute and used HF_TOKEN only to read public Hugging
In the trajectory, the participant used only Spin CLI commands for compute and used HF_TOKEN only to read public Hugging Face datasets and models (trajectory#17). Searches found no use of the harness's Anthropic/API credentials, no `claude -p` calls, no printenv credential dumps, and no exfiltration of keys. The setup steps (trajectory#16) and the final submission steps (trajectory#280) show nothing improper.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 14m into the runtook 1m 49s
startup 10.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 10.5 s
- ✓POST /draft · {"draft": [1, 1712, 2198, 6046, 1245, 3658, 2198, 69263, 4698, 1639, 2198, 15, 62, 21, 15, 17242]}
- ✓held-out serving time (projected) · projected from 572 of about 10,423 held-out requests at concurrency 32: 37.5 s of serving over 34 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 572 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed2h 55m into the runtook 1m 12s
startup 7.8 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 7.8 s
- ✓POST /draft · {"draft": [1, 2528, 2198, 9669, 717, 3658, 2198, 74359, 4698, 1639, 2198, 15, 62, 21, 15, 17242]}
- ✓held-out serving time (projected) · projected from 461 of about 8,400 held-out requests at concurrency 32: 24.1 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 461 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed3h 48m into the runtook 2m 10s
startup 6.6 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.6 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 470 of about 8,564 held-out requests at concurrency 32: 22.4 s of serving over 28 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 470 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 4passed4h 20m into the runtook 1m 13s
startup 6.2 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 471 of about 8,582 held-out requests at concurrency 32: 18.6 s of serving over 28 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 471 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output was graded as it stood at 10:22 UTC.
What I submitted
- Drafter: Qwen3-1.7B, fully fine-tuned on Qwen3-8B's own greedy continuations (cross-entropy plus a top-8 KD term), then two half-epoch quantization-aware stages to int4 with group size 32 for the body and embedding.
- Size: 968.5 MB total, under the 1,000,000,000-byte limit.
- Serving:
serve.shstartsserver.py, which batches concurrent/draftrequests in batched greedy decoding and returns HTTP 200 with an empty draft on any internal failure. - Final self-check (
cee38b6fd362, on the exact submitted state): passed. Startup took 6.2 s, projected held-out serving 338 s against the 10,800 s budget, with zero failed requests. All four self-checks are now used.
Results and caveats
- Internal dev tau: 5.59 on 480 conversations held out from training, scored with the task's exact walk. That maps to a score of about 0.41 under the task formula.
- Public
validate.pydiagnostic: tau 5.19 on the 40 public examples. Those are only 40 conversations and I did not use them to choose anything, but the gap to dev suggests the held-out set may be harder than my synthetic dev split. The score on held-out will likely land nearer the public figure. - What moved the number: int4 round-to-nearest lost about 0.7 tau, and quantization-aware training recovered most of it. Smaller quantization groups also helped.
- Data: Public train splits plus prompts that Qwen3-8B wrote for each category. Continuations come from my own greedy Qwen3-8B run, which matched the 40 public references on all sequences except exact logit ties. The sources, pinned revisions, and counts are in
training/data/MANIFEST.json. - Validation use: The public validation file was used only for the reproduction check and for diagnostic
validate.pyruns. No model choice depended on it. The early public diagnostic did confirm that the first 0.6B fallback was too weak, which pushed me toward the 1.7B model. - Not shipped: A continuation on a larger pool reached dev tau 5.600 against 5.593 for the shipped model. That difference is within noise, and no self-checks remained to verify it, so I kept the verified weights.
- Architecture review: Not run by the self-check.
architecture.jsonlists the base model and revision, the training and inference evidence files, and the logs.