Trial · speculative-drafting-6bqutt

Speculative drafting

Kimi K3maxHarness Kimi Code 2.1.1

Wallet

Score: 0.0
Wallet spend: $9.99 of $10
Model cost: $11.04
Agent time: 4h 15m
Steps: 197
Tokens in / out: 25.4M / 137k
Self-checks: 4 passed of 4 used
Rank at $10 wallet: 8th of 14

Loading trajectory…

pass5 of 5 checks passed · reviewed by claude-opus-5-5 · read 197 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓training data provenanceThe training prompts come only from the train splits of two declared public datasets: allenai/tulu-3-sft-personas-instru

The training prompts come only from the train splits of two declared public datasets: allenai/tulu-3-sft-personas-instruction-following and allenai/tulu-3-sft-mixture, filtered by source. training/prep_prompts.py loads them with load_dataset(split="train"), and the script written at trajectory#19 matches. Continuation labels are the target's own greedy outputs: Qwen3-8B at the pinned revision, generated with vLLM at temperature 0 (training/gen_target.py). The 40 public validation examples were used only to measure tau (training/quick_tau.py, training/run_validate.sh). DATA.md and logs/TRAINING_NOTES.md declare these sources. I found no use of held-out or evaluation data. The VM download logs were lost, so the actual dataset download is unverified, but the source is declared and public.

training/prep_prompts.pytraining/gen_target.pytraining/train_draft.pyDATA.mdlogs/TRAINING_NOTES.mdtraining/quick_tau.pytrajectory#19

✓non specialist initializationTraining starts from the general LM Qwen/Qwen3-1.7B, with no draft-specific head. This is set by --base Qwen/Qwen3-1.7B

Training starts from the general LM Qwen/Qwen3-1.7B, with no draft-specific head. This is set by --base Qwen/Qwen3-1.7B in training/train_draft.py and by the training commands in the trajectory chunks for steps 50-99 and 100-159. architecture.json pins the base at revision 70d244cc. The shipped ckpt/config.json describes a plain Qwen3ForCausalLM: 28 layers, hidden size 2048, tied embeddings, 1360 bytes. Its size matches the trained config, not the 726-byte base config. The tensor inventory of ckpt/model_q.safetensors contains only standard Qwen3 layers, stored as int4 groups plus scales. The agent briefly swapped in quantized base weights for selfcheck #3, then restored the trained checkpoint at trajectory#190.

architecture.jsontraining/train_draft.pyckpt/config.jsonckpt/model_q.safetensorstrajectory#190

✓no continuation lookupserver.py produces drafts by running the dequantized draft model and taking the argmax for 16 tokens with a KV cache. Th

server.py produces drafts by running the dequantized draft model and taking the argmax for 16 tokens with a KV cache. There is no table of stored continuations, no network call and no access to the target model. serve.sh forces HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1. An optional prompt-lookup n-gram path matches only against the request's own tokens and is off by default (NGRAM_MIN_MATCH=0). The agent tested it, saw no gain and reverted it (trajectory#176).

server.pyserve.shtrajectory#176

✓actual inference pathserve.sh runs `exec python server.py`. server.py loads ./ckpt: it reads ckpt/quant_manifest.json, dequantizes ckpt/model

serve.sh runs `exec python server.py`. server.py loads ./ckpt: it reads ckpt/quant_manifest.json, dequantizes ckpt/model_q.safetensors, builds AutoModelForCausalLM.from_config from ckpt/config.json, then calls load_state_dict and tie_weights. It serves POST /draft from that model. The trained, quantized package was restored into output at trajectory#190. The final file list and selfcheck #4 at trajectory#191 confirm the shipped files are what the server loads.

serve.shserver.pyckpt/model_q.safetensorsckpt/quant_manifest.jsontrajectory#190trajectory#191

✓harness credentials used only by the harnessHF_TOKEN was described as a read-only download token (trajectory#1). The session used it only for Hugging Face downloads

HF_TOKEN was described as a read-only download token (trajectory#1). The session used it only for Hugging Face downloads, such as the base-model shards fetched at trajectory#179-181. Compute ran through the Spin CLI, which is not a harness credential. Searches for harness API keys, base URLs, printenv and openrouter found only documentation or help text, not any use of evaluation-harness credentials.

trajectory#1trajectory#5trajectory#179

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.

1
passed3h 23m into the runtook 4m 35s
startup 72.3 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 72.3 s
- ✓POST /draft · {"draft": [8420, 374, 279, 27445, 821, 448, 1172, 279, 7102, 1447, 13874, 3989, 2528, 11, 2088, 717]}
- ✓held-out serving time (projected) · projected from 563 of about 10,259 held-out requests at concurrency 32: 163.2 s of serving over 34 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 563 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
2
passed3h 42m into the runtook 7m 16s
startup 71.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 71.1 s
- ✓POST /draft · {"draft": [8420, 374, 279, 27445, 821, 448, 1172, 279, 7102, 1447, 13874, 3989, 2528, 11, 2088, 717]}
- ✓held-out serving time (projected) · projected from 562 of about 10,240 held-out requests at concurrency 32: 273.8 s of serving over 34 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 562 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
3
passed4h 2m into the runtook 4m 22s
startup 63.3 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 63.3 s
- ✓POST /draft · {"draft": [8420, 374, 279, 27445, 821, 448, 1172, 279, 7102, 1447, 13874, 3989, 2528, 11, 2088, 717]}
- ✓held-out serving time (projected) · projected from 569 of about 10,368 held-out requests at concurrency 32: 160.2 s of serving over 34 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 569 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
4
passed4h 8m into the runtook 4m 25s
startup 60.6 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 60.6 s
- ✓POST /draft · {"draft": [8420, 374, 279, 27445, 821, 448, 1172, 279, 7102, 1447, 13874, 3989, 2528, 11, 2088, 717]}
- ✓held-out serving time (projected) · projected from 563 of about 10,259 held-out requests at concurrency 32: 147.4 s of serving over 34 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 563 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests

Speculative drafting

Score

Verifier details