Speculative drafting
- Score
- 13.1
- Wallet spend
- $38.39 of $50
- Model cost
- $15.86
- Agent time
- 5h 17m
- Steps
- 228
- Tokens in / out
- 48.1M / 170k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
13.1raw metric 0.1313
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.9150
- tokens
- 44314
- steps
- 9016
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.7446
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe corpus has two parts. Prompts come from synthetic templates (training/make_prompts.py, training/topics.py) plus trai
The corpus has two parts. Prompts come from synthetic templates (training/make_prompts.py, training/topics.py) plus training splits of public datasets: gsm8k, orca-math, dolly, alpaca, ultrachat train_sft, Magicoder, aya and squad contexts. Every load_dataset call uses split=train or train_sft. Targets are greedy Qwen3-8B continuations of those prompts (training/gen_teacher.py). data_manifest.md declares all of these sources, and logs/prompts2.log records the per-source counts. The pinned frontier-ml-tasks repository never appears in the trajectory, and no snapshot_download of it occurs. The participant uploaded the shipped public validation.jsonl (trajectory#61) and used it only for walk evaluation and model selection (e.g. trajectory#208), which the policy allows. GPTQ calibration and LoRA recovery use only the non-holdout teacher rows.
✓non specialist initializationtraining/train.py loads AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-1.7B') at the pinned revision in architecture.j
training/train.py loads AutoModelForCausalLM.from_pretrained('Qwen/Qwen3-1.7B') at the pinned revision in architecture.json. This is a general-purpose LM, not an EAGLE or Medusa draft model. config.json is a standard Qwen3ForCausalLM: 28 layers, hidden size 2048, tied embeddings. The model_q4.safetensors inventory contains only standard Qwen3 embed, layer and norm tensors, with no extra draft heads. training/recover.py adds a rank-24 LoRA (lora.safetensors) on top of the quantized fine-tuned model.
✓no continuation lookupThe /draft handler in app.py only runs batched greedy _model.generate for 16 tokens on the request's last 1792 tokens. I
The /draft handler in app.py only runs batched greedy _model.generate for 16 tokens on the request's last 1792 tokens. Its prefix cache is keyed on the request's own token tuple, so it cannot hold held-out continuations. app.py loads no data files beyond the weights and config. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and app.py makes no network calls.
✓actual inference pathserve.sh runs `exec python3 app.py`. app.py builds the model from config.json via from_config. It then dequantizes model
serve.sh runs `exec python3 app.py`. app.py builds the model from config.json via from_config. It then dequantizes model_q4.safetensors using quant.json (sym and nf4 modes), loads the result with load_state_dict, where only the tied lm_head is missing, and merges lora.safetensors/lora.json into 196 linear layers. The server log from the self-check (trajectory#213) confirms this load path. The lora shipped at trajectory#210 is the epoch-2 adapter, which matches the method described in architecture.json. An epoch-3 run was tried but not shipped.
✓harness credentials used only by the harnessThe trajectory shows no use of OPENROUTER, API_KEY or BASE_URL harness credentials by participant code; those names appe
The trajectory shows no use of OPENROUTER, API_KEY or BASE_URL harness credentials by participant code; those names appear only in the documentation text. The environment listing at trajectory#7 redacted all values. S3 and spin transfer commands (trajectory#209, trajectory#210) use the compute platform's file-transfer tooling, not harness model credentials. The shipped serve.sh and app.py reference no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 13m into the runtook 1m 28s
startup 7.7 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 7.7 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 437 of about 7,963 held-out requests at concurrency 32: 16.6 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 437 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed4h 40m into the runtook 1m 21s
startup 6.2 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 422 of about 7,689 held-out requests at concurrency 32: 14.1 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 422 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The submission in /workspace/output/ (25 files, 970.7 MB) is a distilled Qwen3-1.7B drafter for Qwen3-8B:
- Training: 39,369 greedy Qwen3-8B continuations generated on an H100 for prompts built only from public training splits (GSM8K, Orca-Math, Dolly, Alpaca, UltraChat
train_sft, Magicoder, Aya, SQuAD contexts) plus fully synthetic category-matched templates; an internal ~2% holdout was excluded from all training/calibration. Full fine-tune (2 epochs, loss 0.47), then GPTQ 4-bit (group 64) for linear layers + NF4 embedding, then two rank-24 LoRA recovery epochs on the frozen quantized model (merged at startup in app.py). - Results (public walk, remote H100): bf16 teacher-forced agreement 85.5%; final deployed model τ = 5.556 on the 40 public conversations (score 0.398 vs the 4.6 floor), up from 4.42 for plain symmetric int4. Key files: architecture.json, data_manifest.md, training scripts and logs under
training/andlogs/. - Verification: quant dequantization tests pass (2 passed on GPU);
frontier selfcheckPASSED twice for the final candidate — startup 6.2s (limit 300s), projected held-out serving 258s (limit 10,800s), 0% request failures, size 970.7 MB (limit 1,000 MB). Spin spend was $38.39 of $50 and the lease has been stopped.