Speculative drafting
- Score
- 29.5
- Wallet spend
- $82.67 of $200
- Model cost
- $199.70
- Agent time
- 9h 7m
- Steps
- 200
- Tokens in / out
- 46.0M / 262k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 3rd of 14
Score
29.5raw metric 0.2954
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.3090
- tokens
- 44314
- steps
- 8347
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 6.2171
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceDATA_MANIFEST.md declares all prompt sources, and download commands are listed. The public sources are the train split o
DATA_MANIFEST.md declares all prompt sources, and download commands are listed. The public sources are the train split of allenai tulu-3-sft-mixture, the train_sft split of ultrachat_200k and the train split of Magicoder-OSS-Instruct-75K. The other source is synthetic prompts generated by Qwen3-8B. scripts/prep_prompts.py and scripts/prep_prompts2.py call load_dataset only on these train splits. Training targets are Qwen3-8B greedy continuations generated with vLLM (resp_*.jsonl). scripts/train.py reads only those jsonl files. The logs for the selected lineage name only those data files: train_main uses resp_public_A, resp_public_B and resp_synth; train_cont uses resp_public2 and resp_synth2; cont2q_qat uses public3, synth3, public2 and synth2. The public validation.jsonl was used only by scripts/eval_tau.py to measure tau of finished checkpoints, never as training data. The batch-4 response logs on machine F were lost, but batch 4 was used only by the unselected cont3q run. I found no access to hidden or held-out data.
✓non specialist initializationarchitecture.json declares Qwen/Qwen3-1.7B at a pinned revision as the base. scripts/setup_gpu2.sh downloads only the ge
architecture.json declares Qwen/Qwen3-1.7B at a pinned revision as the base. scripts/setup_gpu2.sh downloads only the general Qwen3 models (8B, 1.7B and 0.6B) at pinned revisions. logs/E/train_main_log.txt shows model=/models/qwen3-1.7b. Later stages start from the earlier ones: train_cont from /work/ckpt_main/final, and cont2q_qat from /work/ckpt_cont/final. The shipped tensor shapes match Qwen3-1.7B (hidden size 1024, intermediate size 6144, vocabulary 151936). No EAGLE, Medusa or other pretrained specialist drafter was downloaded or used.
✓no continuation lookupserver.py's /draft endpoint generates drafts by greedy decoding of the drafter model through fast_drafter.py, with draft
server.py's /draft endpoint generates drafts by greedy decoding of the drafter model through fast_drafter.py, with drafter_model.py as an HF fallback. The only cross-request state is a per-conversation KV prefix cache, keyed by a sha1 of the prompt tokens, that holds the model's own key/value tensors. It stores no continuations or answers. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and the bundle contains only weights and config. There are no network calls and no stored response tables.
✓actual inference pathserve.sh runs `python server.py` with DRAFTER_DIR=weights. server.py calls load_drafter, which builds the model from wei
serve.sh runs `python server.py` with DRAFTER_DIR=weights. server.py calls load_drafter, which builds the model from weights/config.json, dequantizes weights/model.q4.safetensors and loads it with a strict load_state_dict. It then wraps that model in FastDrafter, whose custom Qwen3 forward pass uses those same weights. The trajectory shows the shipped weights were copied from s3://spin/ckpt/q4_cont2q (trajectory#183 area, file trajectory/steps-162-183). Their sha256 (b5add731ce44b62fdb2b0f67…) matches the selected q4_cont2q export from logs/E/train_cont2q_qat_log.txt (trajectory#197, trajectory#199).
✓harness credentials used only by the harnessI searched the trajectory and found no use of the harness's ANTHROPIC or other model-API keys or base URLs, and no extra
I searched the trajectory and found no use of the harness's ANTHROPIC or other model-API keys or base URLs, and no extra claude or codex agent sessions. External tools were used only for allowed purposes: Spin machine credentials, s5cmd wallet transfers, an HF token for model and dataset downloads, a curl of the s5cmd GitHub release, and localhost health checks.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 47m into the runtook 2m 29s
startup 7.8 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 7.8 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 500 of about 9,111 held-out requests at concurrency 32: 69.1 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.2 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 500 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed5h 28m into the runtook 2m 35s
startup 8.0 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 8.0 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 441 of about 8,036 held-out requests at concurrency 32: 58.1 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.1 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 441 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed7h 29m into the runtook 13m 28s
startup 8.7 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 8.7 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 433 of about 7,890 held-out requests at concurrency 32: 59.6 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.2 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 433 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
All ~129k training examples are ready. Launching the main run on E (all data, 2 epochs) and a variant on A (synthetic data upsampled 2x, higher LR) so I can pick the better checkpoint later.