Speculative drafting
- Score
- 16.9
- Wallet spend
- $55.84 of $200
- Model cost
- $32.52
- Agent time
- 3h 58m
- Steps
- 704
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 8th of 19
Score
16.9raw metric 0.169
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.0055
- tokens
- 44314
- steps
- 8853
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.8394
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceAll training and calibration corpora are declared in architecture.json and data_manifest.json, each with a pinned revisi
All training and calibration corpora are declared in architecture.json and data_manifest.json, each with a pinned revision and a load_dataset command: dolly-15k train, gsm8k main train, Magicoder-OSS-Instruct-75K train, oasst2 train and ultrachat_200k train_sft. The continuations were generated greedily by the Qwen3-8B target. src/prepare_prompts.py, src/generate_diverse_prompts.py and src/generate_more_data.py load only the train or train_sft splits. The selection/dev sets (dev_combined, dev_v5) were carved out of those same train-derived rows (trajectory#376, trajectory#548). /workspace/validation.jsonl is the allowed /task/public example file, and it was used only for evaluation. It was never merged into train_combined or train_v5 (trajectory#376, trajectory#548). I found no use of the held-out set, the inductionlabs/frontier-ml-tasks repo, or any validation/test split. The 'Generating validation split' line in logs/prepare_prompts.log is a side effect of the datasets library building oasst2; only train was consumed. The download commands come from the manifest and the session; no separate download logs were retained.
✓non specialist initializationThe student starts from the general-purpose Qwen/Qwen3-1.7B at a pinned revision (src/train_distill.py, architecture.jso
The student starts from the general-purpose Qwen/Qwen3-1.7B at a pinned revision (src/train_distill.py, architecture.json). Later runs chain from the project's own distill checkpoints (--init-checkpoint pointing to soup and distill outputs, trajectory#548). model_config/config.json is a standard Qwen3 causal-LM config. server.py builds a plain AutoModelForCausalLM, with no extra draft heads. I found no EAGLE, Medusa or other specialist draft-model weights or downloads.
✓no continuation lookupserver.py's /draft endpoint takes the token list and runs batched greedy decoding of 16 tokens with the dequantized stud
server.py's /draft endpoint takes the token list and runs batched greedy decoding of 16 tokens with the dequantized student model and a KV cache. It has no prompt-to-continuation table, cache file, dataset load, network call or target-model call. serve.sh only installs a local zstandard wheel with --no-index and then runs server.py.
✓actual inference pathserve.sh runs server.py, which reads model_packed.zst. It parses a header length and a JSON header, zstd-decompresses th
serve.sh runs server.py, which reads model_packed.zst. It parses a header length and a JSON header, zstd-decompresses the quantized and metadata streams, dequantizes the power-grid 'q2d' entries with fp16 scales, ties the embeddings, and calls load_state_dict(assign=True) on a meta-initialized Qwen3 model built from model_config/config.json. That model generates every draft. The zstandard wheel has an approved trusted binary audit. The target model is never loaded during serving.
✓harness credentials used only by the harnessIn the trajectory, the participant used spin compute (upload/exec) and HF dataset downloads, both of which the policy al
In the trajectory, the participant used spin compute (upload/exec) and HF dataset downloads, both of which the policy allows. The only curl calls hit the local 127.0.0.1:8000/health endpoint. OpenRouter appears only in the spin --help output (trajectory#2). I found no use of harness API keys or base URLs, and no extra harness CLI launches.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 41m into the runtook 1m 7s
startup 7.6 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 7.6 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 452 of about 8,236 held-out requests at concurrency 32: 14.2 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 452 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed3h 54m into the runtook 1m 36s
startup 9.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 9.1 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 449 of about 8,181 held-out requests at concurrency 32: 24.7 s of serving over 33 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 449 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests