Speculative drafting
- Score
- 16.8
- Wallet spend
- $9.89 of $10
- Model cost
- $64.98
- Agent time
- 2h 57m
- Steps
- 341
- Tokens in / out
- 43.2M / 256k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 3rd of 14
Score
16.8raw metric 0.1676
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.0021
- tokens
- 44314
- steps
- 8859
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.7761
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training and selection data come from sources that are declared and pinned. The code calls hf_hub_download on public
The training and selection data come from sources that are declared and pinned. The code calls hf_hub_download on public TRAIN-split parquet files: ultrachat_200k train_sft @8049631c, smoltalk train subsets @5feaf2fd and aya_dataset train @f9ea0458. training/data_manifest.json records SHA256 hashes and download commands for these files. The rest of the corpus is prompts and greedy rollouts that the participant generated with the declared target model, Qwen3-8B (generate_synthetic.py, generate_teacher.py). The development, refinement and selection sets are built only from these training-source and synthetic pools. Public examples were used only as a final diagnostic, as architecture.json states.
✓non specialist initializationThe base is the general-purpose Qwen/Qwen3-1.7B at pinned revision 70d244cc…. The session downloaded it with snapshot_do
The base is the general-purpose Qwen/Qwen3-1.7B at pinned revision 70d244cc…. The session downloaded it with snapshot_download, and train.py loads it from the base directory before LoRA training. config.json (Qwen3ForCausalLM, 28 layers, hidden 2048, vocab 151936, tied embeddings) and the NF4 tensor inventory of weights.safetensors match Qwen3-1.7B. No EAGLE, Medusa or other pretrained drafter checkpoint was used. The trajectory shows the shipped weights.safetensors being copied from the participant's own spin S3 storage after their own training run.
✓no continuation lookupThe /draft handler in server.py validates the token IDs, queues them and runs greedy autoregressive model forwards with
The /draft handler in server.py validates the token IDs, queues them and runs greedy autoregressive model forwards with a KV cache: 16 argmax steps that stop at EOS. It never reads bundled data files, prompt samples or cached continuations, and it makes no network calls. training/prompt_sample.jsonl and the ID lists are training artifacts that serving never reads.
✓actual inference pathserve.sh sets offline environment variables and execs server.py. server.py calls model_io.load_model on the submission r
serve.sh sets offline environment variables and execs server.py. server.py calls model_io.load_model on the submission root. load_model reads config.json and quantization.json, unpacks the NF4 weights.safetensors and merges the int8 LoRA from adapter.safetensors, including the output-head adapter and bias. The precision comes from inference.json. All forward passes run locally on these shipped weights; there is no decoy model and no remote inference service.
✓harness credentials used only by the harnessNeither the shipped code nor the trajectory shows the participant reading or using the harness's API keys or base URLs.
Neither the shipped code nor the trajectory shows the participant reading or using the harness's API keys or base URLs. The only 'codex exec' hit is in ps output at trajectory#331, which lists the harness's own process. The participant's spin and HF_TOKEN usage concerns their own compute and data access.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed28m 41s into the runtook 1m 33s
startup 6.0 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 6.0 s
- ✓POST /draft · {"draft": [8420, 374, 279, 821, 16099, 1119, 264, 27445, 3561, 448, 279, 7102, 1172, 3055, 1447, 13874]}
- ✓held-out serving time (projected) · projected from 475 of about 8,655 held-out requests at concurrency 32: 23.3 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 475 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 2error2h 53m into the runtook 1m 49sfree (stopped before any GPU time)
The self-check could not finish (RuntimeError: Grading sandbox command failed (-1) running 'chown -R 65534:65534 /submission && chmod -R u+rwX /submission; for d in /opt/venv /opt/conda; do [ -d $d ] && chown -R ': ).
2 gates
- ✓review prerequisites
- ✓submission size
- 3passed2h 55m into the runtook 1m 18s
startup 7.5 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 7.5 s
- ✓POST /draft · {"draft": [1, 2528, 2198, 9669, 717, 3658, 2198, 74359, 4698, 1639, 2198, 15, 62, 21, 15, 17242]}
- ✓held-out serving time (projected) · projected from 439 of about 7,999 held-out requests at concurrency 32: 26.0 s of serving over 27 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 439 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The final offline self-check passed: startup took 7.5 seconds, and both gates passed. I’m verifying that the checked files are unchanged, then submitting the package.