Speculative drafting
- Score
- 2.4
- Wallet spend
- $5.81 of $10
- Model cost
- $0.90
- Agent time
- 2h 37m
- Steps
- 465
- Tokens in / out
- 43.6M / 327k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 9th of 19
Score
2.4raw metric 0.02407
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.6578
- tokens
- 44314
- steps
- 9514
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.4742
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.json names the source of every SFT corpus, and each one is a train split or a public corpus: OpenHermes-2.
data_manifest.json names the source of every SFT corpus, and each one is a train split or a public corpus: OpenHermes-2.5, gsm8k main train, hendrycks_math algebra train, MetaMathQA, CodeAlpaca-20k, Magicoder-OSS-Instruct-75K, the aya_dataset train parquet and euclaise/writingprompts, plus synthetic templates. The labels are Qwen3-8B greedy continuations generated with vLLM. training/build_prompts.py defaults to split='train'. training/build_aya_prompts.py downloads only the aya train parquet. AWQ calibration in training/awq_export.py uses dev_data.jsonl, which is the synthetic seed-999 prompts from training/build_dev_ml.py. The trajectory has no access to the held-out set, the inductionlabs/frontier-ml-tasks repository, or any public dataset's validation or test split. The shipped /task/public/validation.jsonl was uploaded (step 68) and used only for diagnostic eval and the AWQ alpha sweep (around step 433), which the operator policy allows for the public examples. The corpus downloads are declared, but apart from the model downloads in training/logs/download.log no download log was retained for them, so their provenance is recorded as unverified.
✓non specialist initializationThe drafter starts from the general-purpose open-weights model Qwen/Qwen3-1.7B at a pinned revision (architecture.json,
The drafter starts from the general-purpose open-weights model Qwen/Qwen3-1.7B at a pinned revision (architecture.json, training/download_models.sh), not from EAGLE, Medusa or any other specialist drafter. training/train_sft.py calls from_pretrained on the downloaded Qwen3-1.7B directory. training/logs/download.log and trajectory steps 50 and 232 record hf downloads of Qwen/Qwen3-1.7B. weights/config.json matches the Qwen3-1.7B shape (hidden 2048, 28 layers, tied embeddings). The weights/quant.safetensors inventory holds only quantized embed/layer tensors and norms, with no extra draft heads.
✓no continuation lookupserve.py sends each request's tokens through a dynamic batcher to Runner.draft_batch in drafter.py, which does a prefill
serve.py sends each request's tokens through a dynamic batcher to Runner.draft_batch in drafter.py, which does a prefill and greedy argmax decoding of up to 16 tokens with the int4 Qwen3 model. There is no table keyed on prompts or continuations, and the server makes no network calls. weights/quant.safetensors holds only model tensors (q/s/si per layer, embed.q/s, norms), with no stored continuations.
✓actual inference pathserve.sh runs 'exec python -u serve.py' with WEIGHTS_DIR pointing at the shipped weights directory. serve.py builds the
serve.sh runs 'exec python -u serve.py' with WEIGHTS_DIR pointing at the shipped weights directory. serve.py builds the engine from drafter.load_int4(weights), which unpacks the int4 groupwise tensors in weights/quant.safetensors as described by weights/quant_meta.json. It applies the AWQ inverse scales, loads them into the qwen3_min.py Qwen3 module configured by weights/config.json, and serves POST /draft from Runner.draft_batch. The final validation of the submitted server (trajectory step 456) ran through this same path.
✓harness credentials used only by the harnessNo trajectory step reads, copies or uses harness API keys or base URLs: searches for ANTHROPIC, API_KEY and BASE_URL fou
No trajectory step reads, copies or uses harness API keys or base URLs: searches for ANTHROPIC, API_KEY and BASE_URL found nothing. The participant's credential use was limited to its own HF_TOKEN for public model downloads (step 232) and the spin compute CLI and SSH keys for its rented GPU machines (steps 50 and 68). These are participant resources, not harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 36m into the runtook 1m 43s
startup 10.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 10.1 s
- ✓POST /draft · {"draft": [8420, 374, 279, 821, 16099, 1119, 27445, 3561, 448, 279, 7102, 1172, 3055, 1447, 73594, 18104]}
- ✓held-out serving time (projected) · projected from 498 of about 9,074 held-out requests at concurrency 32: 35.0 s of serving over 32 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 498 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state:
Submission (/workspace/output, 32 files, 990.4 MB, under the 1 GB cap):
serve.sh+ custom torch-only Qwen3 runner (serve.py,drafter.py,qwen3_min.py) with batched in-place KV decodeweights/— Qwen3-1.7B distilled on Qwen3-8B greedy continuations, int4 groupwise (g32 linears / g16 embedding) with AWQ-style α=0.25 activation scalingarchitecture.json(baseQwen/Qwen3-1.7B@70d244cc…, target @b968826d…, 31 evidence files),data_manifest.json,training/scripts + logs
Measured results
- Public walk over the shipped layout: tau 4.891 (teacher-forced 4.918), score ≈ 0.12
- Serving: 1,796 requests at concurrency 32 in 160 s → projected held-out 810 s of the 10,800 s budget; 0 failed requests
frontier selfcheck: all gates pass — size, startup 10.1 s,/draft0.54 s, projected serving 638 s, 0% failures, healthy after load
Submission: accepted at 2026-10-20 22:13:44 UTC.