Speculative drafting
- Score
- 25.8
- Wallet spend
- $191.58 of $200
- Model cost
- $184.70
- Agent time
- 9h 48m
- Steps
- 895
- Tokens in / out
- 120.0M / 669k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 4th of 14
Score
25.8raw metric 0.2584
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.2202
- tokens
- 44314
- steps
- 8489
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 6.1011
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.json declares every corpus: the public train splits of ultrachat_200k (train_sft), alpaca (train) and Magi
data_manifest.json declares every corpus: the public train splits of ultrachat_200k (train_sft), alpaca (train) and Magicoder's decontaminated training jsonl, plus synthetic prompts and dialogues generated with Qwen3-8B (training/synthetic_prompts.py, training/growth_dialogues.py, training/generate_targets.py). All response targets were regenerated by the Qwen3-8B teacher. training/download_data.py downloads the exact training shard filenames with pinned hf_hub_download calls and logs sha256 values, and logs/download_data.log records those downloads. Selection in training/select_final.py and the dev rows in training/train.py come from a hash-reserved subset (SHA256(id) % 128 == 0) of the training sources. The benchmark's validation.jsonl was used only by the harness's own validate.py run (trajectory#880), never for training or selection. No forbidden partition or benchmark task repo was accessed.
✓non specialist initializationarchitecture.json declares Qwen/Qwen3-1.7B at revision 70d244cc… as the base. training/download_data.py fetches it with
architecture.json declares Qwen/Qwen3-1.7B at revision 70d244cc… as the base. training/download_data.py fetches it with a pinned snapshot_download. logs/initial_training.log shows the first training run starting from /workspace/draftwork/Qwen3-1.7B. Every later stage (focused, expanded, dense, quantized, QAT) starts from a previous in-house checkpoint, as the initialization records in logs/*_training.log show, ending at dense_diverse_qat/step3500. Qwen3-1.7B is a general-purpose language model, not a pretrained speculative drafter. No EAGLE, Medusa or other drafter checkpoint was downloaded or loaded.
✓no continuation lookupserver.py runs model.generate(do_sample=False, max_new_tokens=16) on only the request's token list. The model comes from
server.py runs model.generate(do_sample=False, max_new_tokens=16) on only the request's token list. The model comes from model_io.load_model on local files. There is no network access, cache of benchmark continuations, teacher lookup, retrieval table or hint injection. serve.sh sets offline environment variables and execs server.py.
✓actual inference pathserve.sh starts server.py, which calls model_io.load_model(ROOT). That function builds Qwen3 from config.json and quanti
serve.sh starts server.py, which calls model_io.load_model(ROOT). That function builds Qwen3 from config.json and quantization.json, gunzips weights.safetensors.gz, dequantizes the packed 4-, 5- and 6-bit groups, adds the low-rank int8 corrections and ties the embeddings. The /draft handler uses that model's generate output. At trajectory#879 the shipped weights were copied from the final selected checkpoint, and at trajectory#878 their sha256 values match those recorded in architecture.json. The archive itself had no inventory or trusted audit (it was recorded as uninspected), but the loading path and recorded hashes are consistent, and nothing suggests a different serving model.
✓harness credentials used only by the harnessThe trajectory shows the participant using only its own tooling: vast.ai workers, Spin/s5cmd storage transfers, HF downl
The trajectory shows the participant using only its own tooling: vast.ai workers, Spin/s5cmd storage transfers, HF downloads, vLLM generation and training scripts. No step reads, exports or uses the harness's model-API credentials, and no extra claude or codex agent sessions were launched. Validation was run through the harness's provided validate.py (trajectory#880).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed52m 17s into the runtook 3m 47s
startup 6.2 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /draft · {"draft": [8420, 374, 279, 14892, 32247, 16099, 1119, 27445, 448, 7102, 1172, 3055, 1447, 334, 2528, 11]}
- ✓held-out serving time (projected) · projected from 552 of about 10,058 held-out requests at concurrency 32: 18.3 s of serving over 33 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 552 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed3h 2m into the runtook 1m 21s
startup 6.0 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.0 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 469 of about 8,546 held-out requests at concurrency 32: 16.3 s of serving over 33 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 469 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed9h 47m into the runtook 1m 24s
startup 14.4 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 14.4 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 459 of about 8,364 held-out requests at concurrency 32: 16.1 s of serving over 31 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 459 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Submitted the trained drafter in /workspace/output/.
- Public mean acceptance: 5.19, zero failed requests.
- Offline H100 self-check passed; startup 14.4 seconds.
- Total submission size: 977,573,614 bytes.