Speculative drafting
- Score
- 0.0
- Wallet spend
- $18.73 of $200
- Model cost
- $0.86
- Agent time
- 5h 22m
- Steps
- 309
- Tokens in / out
- 31.2M / 290k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 14th of 19
Score
0.0raw metric 0
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.3057
- tokens
- 44314
- steps
- 10292
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.0858
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe shipped drafter is runs/final2. It was distilled on three declared corpora: synthetic template prompts, train splits
The shipped drafter is runs/final2. It was distilled on three declared corpora: synthetic template prompts, train splits of public HF datasets, and prompts written by Qwen3-8B itself, all answered by the pinned target. These are declared in architecture.json and evidence/data_manifest.md. build_prompts.py's _ds uses split="train" only, and gen_prompts_llm.py uses no dataset rows. The download commands are retained but were not independently re-verified. Searches found no use of the held-out set or the inductionlabs/frontier-ml-tasks repo. /task/public/validation.jsonl was used only as a shipped-public-example evaluation proxy, including the final2-vs-final4 comparison, which the policy allows. A late extra run (final4) used the OpenHermes-2.5 train split. It scored worse and was rejected, not shipped (trajectory#293-305), and it was a train split in any case.
✓non specialist initializationevidence/train.py loads AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-1.7B', revision 70d244cc...). train_final.log r
evidence/train.py loads AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-1.7B', revision 70d244cc...). train_final.log records loading that model (1.721B params), and train_final2.log continues from runs/final. config.json is a standard Qwen3ForCausalLM with the 1.7B shapes (28 layers, hidden 2048, tied embeddings). The model_int4.safetensors inventory has only standard Qwen3 layer tensors (packed int4 plus scales and norms), with no EAGLE/Medusa heads. The trajectory shows `hf download Qwen/Qwen3-1.7B` from the official repo.
✓no continuation lookupserver.py serves POST /draft by validating the tokens and running batched greedy argmax decoding of 16 tokens with the Q
server.py serves POST /draft by validating the tokens and running batched greedy argmax decoding of 16 tokens with the Qwen3 model and a KV cache, stopping at EOS. It has no lookup tables, cached continuations, network calls or alternate response paths. serve.sh sets offline environment variables.
✓actual inference pathserve.sh execs server.py --model-dir with the package directory. load_model reads quant_meta.json and config.json, build
serve.sh execs server.py --model-dir with the package directory. load_model reads quant_meta.json and config.json, builds Qwen3ForCausalLM, dequantizes model_int4.safetensors (int4 group-64) to bf16, loads the norms and ties lm_head. The Engine then uses that model for every /draft response. The tensor inventory matches the shapes the loader expects. The selfcheck served the package successfully, and the trajectory shows that final2 is the shipped checkpoint and that the weights were not replaced before submission.
✓harness credentials used only by the harnessThe only credential the participant used was HF_TOKEN, written to hf.env and used for HF downloads on the Spin VM, which
The only credential the participant used was HF_TOKEN, written to hf.env and used for HF downloads on the Spin VM, which the policy allows. The only MLE_JUDGE_API_KEY occurrence is validate.py source text the participant viewed, and the only 'openrouter' occurrence is spin CLI help text. Searches found no use of a harness API key, base URL or anthropic client in the shipped code or the session.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed4h 46m into the runtook 2m 7s
startup 57.3 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 57.3 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 446 of about 8,127 held-out requests at concurrency 32: 14.1 s of serving over 27 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 446 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final state of the work:
| Drafter | Qwen3-1.7B full fine-tuned from 70d244cc…, int4-group64 (914,339,022 B of model files) |
| Training | Stage 1: 2 epochs / 60k target-greedy conversations (35.0M tok); Stage 2: 1 epoch / 74,968 conversations (21.8M tok) including 14,968 prompts written by Qwen3-8B itself |
| Method | CE on continuation tokens (prompt masked) + 0.5 × top-8 forward KL to target log-probs |
| Public validation | tau 5.028, score 0.178 (40 conversations, 1,747 requests) |
| Own dev | tau 6.552 (480 conversations) |
| Serving | startup 57.3 s of 300 s; 0 failed requests; projected held-out serving 257.6 s of 10,800 s |
| Selfcheck | PASSED (run 2a042d4a7dbc), 914.7 MB, digest 965a2b0d… |
What drove the result: the target stack was reproduced exactly (teacher-forced agreement 99.3% with the public references), and the drafter was distilled purely on target-greedy outputs. The main finding was a large distribution gap — 0.874 teacher-forced top-1 on our own dev prompts versus 0.80 on the task's individually-written prompts — which three separate data efforts only partly closed (+0.08 tau from target-written prompts, +0.01 from extra epochs on them, −0.02 from real GPT-4-written OpenHermes prompts). The shipped checkpoint was the best of those on the public metric; the final pass on OpenHermes prompts was measured and rejected rather than shipped.
All provenance is in evidence/data_manifest.md + prompts*.sources.json + logs: public data is train-split only, the public validation conversations were never training targets, and the newly generated prompt corpora are documented as synthetic. The selfcheck also passed the review-prerequisites gate, though the architecture review itself runs at grading time.