Speculative drafting
- Score
- 48.4
- Wallet spend
- $176.61 of $200
- Model cost
- $41.93
- Agent time
- 9h 26m
- Steps
- 483
- Tokens in / out
- 49.2M / 489k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 1st of 14
Score
48.4raw metric 0.4844
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 5.7625
- tokens
- 44314
- steps
- 7690
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 6.6391
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceDeclared sources: public datasets loaded only from train or train_sft splits in collect_public_prompts.py, plus syntheti
Declared sources: public datasets loaded only from train or train_sft splits in collect_public_prompts.py, plus synthetic prompts generated by Qwen3-8B. Every continuation was regenerated by Qwen3-8B (vLLM, temperature 0, top-20 logprobs) in gen_targets.py. The source is declared in DATA_MANIFEST.md and architecture.json. The shipped /task/public/validation.jsonl examples were used only for monitoring (pub_tau) and for exact-duplicate filtering of synthetic prompts. Policy allows this, and the R2 copy is the participant's own upload of that public file (trajectory#112-128). Dev shards were carved from the candidate's own pool. No held-out data, validation/test splits or pinned task repo appear in code or in the session. Some downloads of declared public sources have no retained logs, so those are unverified, but no violation is established.
✓non specialist initializationtrain_kd.py calls AutoModelForCausalLM.from_pretrained on Qwen/Qwen3-1.7B, snapshot-downloaded at the pinned revision 70
train_kd.py calls AutoModelForCausalLM.from_pretrained on Qwen/Qwen3-1.7B, snapshot-downloaded at the pinned revision 70d244cc in setupF.sh. The m1 log shows init from /root/models/Qwen3-1.7B with no resume. The f1 log shows the same init, resumed from init2.pt. Trajectory#347 shows init2.pt is the participant's own m1.pt checkpoint fetched from R2. The weight inventory is a plain Qwen3 transformer: embed, per-layer q/k/v/o/gate/up/down, norms, int4 packed. It has no EAGLE/Medusa heads, and no specialist drafter was referenced.
✓no continuation lookupserver.py validates the token list, truncates the context and calls ENGINE.draft. engine.py runs a pure Qwen3 forward (p
server.py validates the token list, truncates the context and calls ENGINE.draft. engine.py runs a pure Qwen3 forward (prefill plus greedy argmax decode with CUDA graphs, optional beam off by default) over the dequantized weights. There is no corpus, cache of continuations, n-gram table, network call or retrieval in the serving path. serve.sh sets HF offline mode.
✓actual inference pathserve.sh execs server/server.py --model-dir weights, and engine.py loads weights/model.safetensors with quant.json and c
serve.sh execs server/server.py --model-dir weights, and engine.py loads weights/model.safetensors with quant.json and config.json. The tensor inventory (embed.q/.s, per-layer .q/.s, 28 layers) matches the engine's loader and the Qwen3-1.7B config. Trajectory#448 copies the f1 export into the output weights with a sha256 check. Trajectory#445 shows a real-engine walk on that export.
✓harness credentials used only by the harnessThe trajectory shows only Spin and spin transfer, including a refresh of expired R2 credentials via `spin transfer crede
The trajectory shows only Spin and spin transfer, including a refresh of expired R2 credentials via `spin transfer credentials`, which is permitted. No step reads, prints, copies or uses the harness's API keys or base URL; those strings appear only in Spin documentation output. No call to the target-model harness credentials was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 20m into the runtook 27sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed1h 21m into the runtook 1m 16s
startup 12.3 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 12.3 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 485 of about 8,837 held-out requests at concurrency 32: 13.5 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.1 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 485 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 3passed9h 19m into the runtook 1m 29s
startup 13.0 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 13.0 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 427 of about 7,780 held-out requests at concurrency 32: 12.1 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.2 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 427 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
- 4passed9h 25m into the runtook 1m 47s
startup 26.8 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 26.8 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 426 of about 7,762 held-out requests at concurrency 32: 17.3 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.1 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 426 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and the session is over: /workspace/output was submitted at 07:44 UTC, and its score isn't known yet.
The drafter is Qwen3-1.7B, trained by distillation from Qwen3-8B's own greedy continuations and quantized to int4 (968,044,592 B of weights). On the 40 public example conversations it reaches a mean acceptance length of about 5.78, which would score about 0.49 if the held-out set behaves like them. The target is 7.0 for full marks.
The final frontier selfcheck passed: startup was 26.8 s of 300 s and both gates passed. It does not run the architecture review, so I can't say how that will go. To prepare for it, architecture.json, DATA_MANIFEST.md and logs/README.md document the data sources, training runs, final model choice and the evaluation-script fix. The manifest also discloses that the hand-written topic lists were informed by the 40 public examples, whose text was never used as training data.