Speculative drafting
- Score
- 33.8
- Wallet spend
- $45.53 of $50
- Model cost
- $29.41
- Agent time
- 7h 49m
- Steps
- 389
- Tokens in / out
- 36.7M / 350k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 2nd of 14
Score
33.8raw metric 0.3384
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.4121
- tokens
- 44314
- steps
- 8188
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 6.3195
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceDATA_SOURCES.md declares every corpus. The training prompts come from two sources. (1) Qwen3-8B-synthesized prompts made
DATA_SOURCES.md declares every corpus. The training prompts come from two sources. (1) Qwen3-8B-synthesized prompts made by train/gen_prompts.py, using few-shot style taken only from the /task/public prompt text. (2) Train splits of gsm8k, MetaMathQA, CodeAlpaca-20k, aya (non-English), dolly-15k and alpaca-cleaned, each pinned to a revision in train/gen_public_prompts.py. Every load_dataset call there uses a train or train_sft split; ultrachat failed and was not used. The labels are Qwen3-8B greedy continuations plus top-8 logprobs from train/gen_responses.py. Model selection uses a synthetic dev slice; the public validation set is used only for monitoring. The training logs list data dirs that match these sources. The trajectory shows the dataset revision SHAs being pinned through the HF API (trajectory#49, #50). Download logs are only partly retained, so the public-dataset provenance is declared but partly unverified. I found no use of held-out, test or benchmark-answer data.
✓non specialist initializationtrain/train.py initializes with from_pretrained(Qwen/Qwen3-1.7B) at the pinned BASE_REV revision, or with --init from th
train/train.py initializes with from_pretrained(Qwen/Qwen3-1.7B) at the pinned BASE_REV revision, or with --init from the previous stage's checkpoint. train_A_bf16.txt shows init None from Qwen/Qwen3-1.7B. Each later stage chains from that (Q4 init runs/Q3b/final). architecture.json declares Qwen/Qwen3-1.7B at rev 70d244cc. model/config.json and the safetensors inventory match the Qwen3-1.7B shape (28 layers, hidden 2048, vocab 151936, tied embeddings). I found no pre-trained drafter, EAGLE/Medusa head or other specialist initialization in the code or the trajectory.
✓no continuation lookupserve.py runs autoregressive greedy decoding from the dequantized Qwen3 model only. It contains no n-gram or continuatio
serve.py runs autoregressive greedy decoding from the dequantized Qwen3 model only. It contains no n-gram or continuation tables, no cache of reference outputs and no network calls. Warm-up uses synthetic tokens. The model directory holds only config.json, export_meta.json and model.safetensors, with quantized weight tensors (.qw/.sc/.zp plus norms). It contains no lookup data, and optional keep_ids.npy is not shipped.
✓actual inference pathserve.sh execs serve.py offline (HF_HUB_OFFLINE=1). serve.py builds Qwen3ForCausalLM from model/config.json, loads model
serve.sh execs serve.py offline (HF_HUB_OFFLINE=1). serve.py builds Qwen3ForCausalLM from model/config.json, loads model/model.safetensors, dequantizes the int4 groups using export_meta group 64, calls load_state_dict and tie_weights, then serves POST /draft through a batched KV-cache greedy decode of 16 tokens. The shipped safetensors size (941,198,360 bytes) matches the export of runs/Q4/final by train/export.py (trajectory#362). That export was installed as model/ and validated through serve.sh (trajectory#368, #369). The serving process therefore uses these weights.
✓harness credentials used only by the harnessIn the trajectory, HF_TOKEN was used only for Hugging Face model and dataset downloads and HF API revision lookups on sp
In the trajectory, HF_TOKEN was used only for Hugging Face model and dataset downloads and HF API revision lookups on spin machines, and it was redacted in output (trajectory#47, #48, #49). The other tooling used was the spin CLI. Searches found no use of harness API keys, harness base URLs or Anthropic credentials, and no extra harness CLI sessions. ANTHROPIC appears only in a final secret-audit grep.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 5m into the runtook 1m 40s
startup 7.0 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 7.0 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 428 of about 7,799 held-out requests at concurrency 32: 22.4 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 428 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed7h 47m into the runtook 1m 29s
startup 5.7 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.7 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 411 of about 7,489 held-out requests at concurrency 32: 15.7 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 411 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
I submitted the drafter. frontier selfcheck passed on the final folder: startup 5.7 s, a projected 285 s of serving time (limit 10,800 s), 0 failed requests. I could not measure the real score, since the held-out set is not available to me.
- Model: Qwen3-1.7B, fully fine-tuned to predict Qwen3-8B's greedy continuations, then trained for int4 (group-64) quantization.
serve.pyloads it as bf16 and drafts 16 greedy tokens perPOST /draft. - Size: 941.8 MB in total, under the 1,000,000,000-byte limit.
- Public validation: mean acceptance length tau 6.016 on the 40 public examples, a diagnostic score of 0.59 (the previous model scored 5.992). On my held-out synthetic dev slice, tau was 6.036.
- Beam search: it made drafts worse (5.79–5.81 versus 6.03 for greedy on the public set), so serving stays greedy.
- Weight averaging: averaging the last checkpoints was inconclusive, so I kept the unmodified last-stage weights.
- Documentation:
DATA_SOURCES.mdandarchitecture.jsonlist every data source and the training scripts and logs. The machine logs for data rounds 2 and 4 were lost when those machines were released, andDATA_SOURCES.mdsays so.
The session is now over and /workspace/output is being graded.