Speculative drafting
- Score
- 0.0
- Wallet spend
- $8.91 of $10
- Model cost
- $0.79
- Agent time
- 2h 4m
- Steps
- 324
- Tokens in / out
- 31.5M / 356k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 11th of 19
Score
0.0raw metric 0
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.5025
- tokens
- 44314
- steps
- 9842
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.2175
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancetraining/data_manifest.json declares where the training data came from. The prompts were made by the submission's own te
training/data_manifest.json declares where the training data came from. The prompts were made by the submission's own template generators (training/make_prompts.py, training/make_prompts2.py), and neither loads any dataset. The target labels are greedy continuations from Qwen3-8B, produced by training/gen_target.py with vLLM. The public example conversations were used only for format checks and for a final validation report (training/logs/validate_public_report.json). They did not feed training or model selection. Searches for evaluation-data loading (load_dataset, frontier-ml-tasks, validation ids) found nothing in the training code. The shipped data samples (training/data/sample_train.jsonl and training/data/val_sup.jsonl) are self-generated, with ids of the form gen-.
✓non specialist initializationarchitecture.json declares the base as Qwen/Qwen3-1.7B, a general pretrained language model, at revision 70d244cc. train
architecture.json declares the base as Qwen/Qwen3-1.7B, a general pretrained language model, at revision 70d244cc. training/download_model.sh and its log show that revision being downloaded with snapshot_download. training/train_drafter.py loads the starting weights with AutoModelForCausalLM.from_pretrained(--base). The trajectory shows the chain: Qwen3-1.7B, then stage B, then stage F (base /root/out/B), then stage G (base /root/out/F). G was then quantized to G_q32 and linked as the shipped model. The shipped weights.safetensors holds only standard Qwen3-1.7B decoder tensors in quantized form, with no pretrained draft heads. Nothing indicates a pretrained speculative drafter or specialist checkpoint was used.
✓no continuation lookupThe serving path reads only model/config.json and model/weights.safetensors. server.py routes POST /draft through a Batc
The serving path reads only model/config.json and model/weights.safetensors. server.py routes POST /draft through a Batcher to DraftModel.draft, which in drafter.py runs a full Qwen3 forward pass with greedy argmax over 16 tokens. The server makes no network calls, loads no data files and contains no cache or lookup table keyed on prompts or continuations. The training jsonl files are never opened at serve time.
✓actual inference pathserve.sh execs server.py with --model-dir set to the shipped model directory, and the environment is offline (HF_HUB_OFF
serve.sh execs server.py with --model-dir set to the shipped model directory, and the environment is offline (HF_HUB_OFFLINE). drafter.py loads the int4 group-32 weights from model/weights.safetensors, dequantizes them to bf16 on the GPU and runs a custom Qwen3 forward pass with a KV cache to produce every draft token. The tensor inventory matches the names and shapes the loader expects and model/config.json. No decoy checkpoint or alternate generation path exists, and the target model is never loaded at serving time.
✓harness credentials used only by the harnessThe trajectory shows no use of harness credentials by the participant. Searches found no ANTHROPIC key references and no
The trajectory shows no use of harness credentials by the participant. Searches found no ANTHROPIC key references and no outbound API calls. BASE_URL appears only in help text. At step 86 the participant read only a timestamp from its own session log. All curl calls went to the local /health and /draft endpoints of the participant's own server, and no other agent CLI was used.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 53m into the runtook 1m 17s
startup 2.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.1 s
- ✓POST /draft · {"draft": [1, 2528, 2198, 9669, 717, 3658, 2198, 74359, 4698, 1639, 2198, 15, 62, 21, 15, 17242]}
- ✓held-out serving time (projected) · projected from 493 of about 8,983 held-out requests at concurrency 32: 11.1 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.7 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 493 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2passed2h 3m into the runtook 1m 4s
startup 2.2 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.2 s
- ✓POST /draft · {"draft": [1, 2528, 2198, 9669, 717, 3658, 2198, 74359, 4698, 1639, 2198, 15, 62, 21, 15, 17242]}
- ✓held-out serving time (projected) · projected from 493 of about 8,983 held-out requests at concurrency 32: 11.3 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.7 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 493 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Final selfcheck passed on the exact directory being submitted. Submitting: