Speculative drafting
- Score
- 0.0
- Wallet spend
- $0.00 of $10
- Model cost
- $5.41
- Agent time
- 34m
- Steps
- 435
- Tokens in / out
- 47.1M / 133k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 8th of 14
Score
0.0raw metric 0
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 1.1180
- tokens
- 44314
- steps
- 39638
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 1.2421
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the drafter was trained. prepare_weights.py only downloads Qwen/Qwen3-0.6B-FP8 at a pinned revision. It then
Nothing in the drafter was trained. prepare_weights.py only downloads Qwen/Qwen3-0.6B-FP8 at a pinned revision. It then drops the redundant tied lm_head.weight tensor, shifts the data offsets to match, and sets dequantize=true in config.json. No training, fine-tuning or calibration code is shipped. The session shows no load_dataset or snapshot_download calls, no access to the pinned frontier-ml-tasks repository and no 'private' paths. The only task data it read was /task/public/validation.jsonl, the shipped public examples, which the policy explicitly allows, and that was used only for sanity tests. No training corpus is involved.
✓non specialist initializationThe declared base is Qwen/Qwen3-0.6B-FP8 at revision e5be0803..., a general pretrained language model, not a speculative
The declared base is Qwen/Qwen3-0.6B-FP8 at revision e5be0803..., a general pretrained language model, not a speculative drafter. The weight inventory matches a plain Qwen3-0.6B FP8 layout: 28 layers (0 to 27), embed_tokens, model.norm, FP8 weights with weight_scale_inv, and no extra draft or MTP heads. The file is 751,810,665 bytes. The session shows the weights were written directly from the cached Qwen3-0.6B-FP8 blob by the same stripping procedure, and the resulting size matches exactly.
✓no continuation lookupThe /draft handler in server.py takes the tokens from the request, left-pads them, and runs greedy model.generate for 16
The /draft handler in server.py takes the tokens from the request, left-pads them, and runs greedy model.generate for 16 new tokens with the bundled model. There are no lookup tables, hashes or stored continuations, and the server makes no network calls. Besides the model's config and weights, the only bundled files are vendored wheels, and trusted audits approve them as upstream PyPI wheels. The drafter is not the target model: it is Qwen3-0.6B-FP8, not Qwen3-8B.
✓actual inference pathserve.sh installs the vendored wheels and then runs exec python3 server.py. server.py loads AutoModelForCausalLM from th
serve.sh installs the vendored wheels and then runs exec python3 server.py. server.py loads AutoModelForCausalLM from the model/ directory, the only checkpoint shipped, and every /draft response comes from that model's generate output. There is no alternate path, decoy checkpoint or remote service.
✓harness credentials used only by the harnessIn the session, the OPENROUTER/LiteLLM strings appear only inside the harness's own launch command, which showed up in t
In the session, the OPENROUTER/LiteLLM strings appear only inside the harness's own launch command, which showed up in the output of a ps call. The participant never read, echoed or used harness API keys or base URLs, and never launched other agent CLIs. HF_TOKEN was used only for read-only Hugging Face API and file requests, which is allowed, and spin was used only for compute offers and help.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed19m 31s into the runtook 8m 4s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: INFO:drafter:Using device: cuda INFO:drafter:Loading drafter model from /submission/model... `torc
3 gates
- ✓review prerequisites
- ✓submission size
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: INFO:drafter:Using device: cuda INFO:drafter:Loading drafter model from /submission/model... `torch_dtype` is deprecated! Use `dtype` instead! Traceback (most recent call last): File "/submission/server.py", line 24, in <module> model = AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4806, in from_pretrained raise ValueError( ValueError: Using a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate`. You can install it with `pip install accelerate`
- 2passed31m 14s into the runtook 2m 47s
startup 11.3 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 11.3 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 2180 of about 39,722 held-out requests at concurrency 32: 99.5 s of serving over 76 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 6.1 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 2180 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Summary of Final Submission
-
Model Selection & Footprint Optimization:
- Base Model:
Qwen/Qwen3-0.6B-FP8at revisione5be08033360965ceca7b0ffd72d521a51331ce0. - Tokenizer Compatibility: Exact vocabulary and tokenization alignment with the target
Qwen/Qwen3-8B. - Size Reduction: The raw FP8 checkpoint was 1.06 GB. By taking advantage of Qwen3's tied word embeddings (
tie_word_embeddings: true) and shifting safetensors data offsets, the redundant 311 MBlm_head.weightwas removed without losing any model fidelity. - Total Submission Size: 756,251,412 bytes (~756.3 MB), comfortably within the 1,000,000,000 byte limit.
- Base Model:
-
Serving & Batching Implementation:
- Packaged a high-throughput, low-latency async HTTP server using
aiohttpin server.py. - Designed micro-batching with left-padding to concurrently draft up to 32 requests within a 15ms collection window.
- Configured weight dequantization to native BF16 (
torch.bfloat16) on device initialization to eliminate any offline CUDA compiler or external GEMM runtime dependencies. - Vendored all necessary wheels offline in
/workspace/output/wheels.
- Packaged a high-throughput, low-latency async HTTP server using
-
Validation & Verification:
- Passed
frontier selfcheckon an H100 GPU (NVIDIA H100 80GB HBM3):- Prerequisites & Submission Size: Passed.
- Startup: 11.3 s (limit: 300 s, 96% margin).
- Projected Serving Time: 1,819.5 s (limit: 10,800 s, 83% margin).
- Request Failures: 0% across 2,180 benchmark requests.
- Submitted successfully via
frontier submit --confirm.
- Passed