Speculative drafting
- Score
- 13.7
- Wallet spend
- $31.65 of $200
- Model cost
- $26.78
- Agent time
- 8h 18m
- Steps
- 354
- Tokens in / out
- 33.6M / 290k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 7th of 14
Score
13.7raw metric 0.137
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.9287
- tokens
- 44314
- steps
- 8991
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.7784
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is declared in MANIFEST.md and architecture.json, and the code that built it is shipped. evidence/ge
The training corpus is declared in MANIFEST.md and architecture.json, and the code that built it is shipped. evidence/gen_prompts.py had the pinned target Qwen3-8B (rev b968826d…) write its own prompts via vLLM, and every row is labelled source 'llm-generated'. evidence/gen_continuations.py then produced the target's greedy continuations of those prompts. The 40 shipped public validation examples were used only as style few-shots and for sanity evaluation, which is allowed. The session confirms this: the public file was copied from /task/public (trajectory#20 and #33). The final corpus was conts.jsonl plus conts23.jsonl, minus the holdout (trajectory#225). I found no external dataset loads (load_dataset, gsm8k, HuggingFaceH4 or split= all returned nothing). The gen_prompts.py docstring mentions public train slices, but the code never uses them.
✓non specialist initializationevidence/train.py loads general-purpose Qwen/Qwen3-1.7B at the declared revision 70d244cc… with from_pretrained. The ses
evidence/train.py loads general-purpose Qwen/Qwen3-1.7B at the declared revision 70d244cc… with from_pretrained. The session launched exactly that model to produce draft17b_v2 (trajectory#225). evidence/qat.py fine-tunes from that trained model. The only downloads in the session were Qwen3-8B (the target) and Qwen3-0.6B (trajectory#28), and no EAGLE or other specialist drafter was fetched. The model.int4.safetensors inventory has standard Qwen3 tensors whose shapes fit the 1.7B model (hidden 2048, intermediate 6144, vocab 151936).
✓no continuation lookupserver.py answers /draft only by running the model: a batched, left-padded prefill, then greedy decoding of up to 16 tok
server.py answers /draft only by running the model: a batched, left-padded prefill, then greedy decoding of up to 16 tokens with an EOS stop. It has no lookup tables, no cached reference continuations and no network calls. serve.sh sets offline mode before launching it.
✓actual inference pathserve.sh execs server.py. load_model builds the model from the local config.json with from_config, then loads model.int4
serve.sh execs server.py. load_model builds the model from the local config.json with from_config, then loads model.int4.safetensors and dequantizes the .pack/.scale/.zero tensors using the group size in quant_meta.json. load_state_dict asserts there are no unexpected keys and no missing keys, except a tied lm_head. So the served model is the shipped quantized checkpoint, and the inventory matches that format.
✓harness credentials used only by the harnessIn the session I found no use of harness credentials outside the harness. The participant only read the validate.py/cli.
In the session I found no use of harness credentials outside the harness. The participant only read the validate.py/cli.py flags and ran the public validation harness (trajectory#105, trajectory#106). There was no judge-key or API-key use, no ANTHROPIC or OPENROUTER calls, and no use of the harness base URL; the only hit was Spin CLI help text (trajectory#14). Model downloads used HF access and Spin compute, both of which are permitted (trajectory#28).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed8h 14m into the runtook 3m 38s
startup 28.1 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 28.1 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 468 of about 8,528 held-out requests at concurrency 32: 37.9 s of serving over 29 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 468 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 09:33:47 UTC — the task is complete. /workspace/output is graded as-is (32 files, 943.0 MB, digest f7ede3c2…).
Final submission summary:
- Model: Qwen3-1.7B (rev
70d244cc…) → 2 epochs behavioral cloning on 142k self-generated conversations (target's own greedy continuations over LLM-generated prompts across all 8 categories; the public 40 were never trained on) → 2 rounds of QAT → group-128 int4 (914 MB), dequantized to fp32 at startup. - Live public tau 5.1731 at concurrency 32, deterministic across runs, 0 failed requests, 28s startup, projected held-out serving 690s of the 10,800s budget.
The decisive moment this stretch: the first live walk scored 4.43 — below the 4.6 floor, i.e. score zero — despite the teacher-forced eval saying 5.18. Three serving bugs were responsible, all found and fixed via live validation runs:
- bf16 GEMM rounding varies with batch composition → argmax near-ties flipped under load, derailing 16-token rollouts (−0.74 tau). Serving in float32 with TF32 disabled made drafts batch-invariant and restored the full 5.17.
- No
torch.inference_mode()→ the autograd graph retained all prefill activations, OOMing under load (76 GB of live tensors). num_logits_to_keeprenamed tologits_to_keepin transformers 4.57 → the old name was silently swallowed, computing 13 GB of full-vocab logits per prefill.
Honest score expectation: if the held-out set behaves like the public 40, tau ≈ 5.17 → score ≈ 0.24; if it behaves like my internal generated holdout (tau 5.52), score ≈ 0.38. All contract gates passed selfcheck with 91–100% margin.