Speculative drafting
- Score
- 0.0
- Wallet spend
- $9.75 of $10
- Model cost
- $30.70
- Agent time
- 2h 52m
- Steps
- 76
- Tokens in / out
- 40.2M / 96k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 8th of 14
Score
0.0raw metric 0
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- mean_acceptance_length
- tau
- 4.5807
- tokens
- 44314
- steps
- 9674
- samples
- 200
- draft_tokens
- 16
- tau_macro_over_prompts
- 5.3883
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe calibration and selection data are declared. data_manifest.json and download_data.sh list HuggingFaceTB/smoltalk sub
The calibration and selection data are declared. data_manifest.json and download_data.sh list HuggingFaceTB/smoltalk subsets (split=train) and sahil2801/CodeAlpaca-20k (train), prompts only. Continuations were generated by the Qwen3-8B teacher at a pinned revision, and the manifest says no validation or test split was used. code/prepare_prompts.py calls load_dataset(..., split='train'), and logs/prep.log records 4400 prompts written. code/batch_gen.py and logs/batchgen.log show greedy teacher traces. code/gptq_pack.py calibrates on traces_b and traces_v2, and model/quant_meta.json repeats the train-split sources. The script that produced traces_v2 (logs/gen2.log) is not shipped, but the declared sources cover it. In the remote listing, traces_v2.jsonl has the same size and timestamp as traces.jsonl, which was built from the same prompts.jsonl. The public validation.jsonl was used only for local evaluation, which is allowed. Searches of the trajectory for the held-out repo (inductionlabs/frontier-ml-tasks) found no access. Downloads are not fully logged, so the declared public sources count as unverified, not as a violation.
✓non specialist initializationarchitecture.json declares base_model Qwen/Qwen3-1.7B at a pinned revision. code/gptq_pack.py quantizes /root/models/Qwe
architecture.json declares base_model Qwen/Qwen3-1.7B at a pinned revision. code/gptq_pack.py quantizes /root/models/Qwen3-1.7B. logs/gptq.log reports saved bytes of 968082992, exactly the size of the shipped model/model.safetensors. That checkpoint contains only standard Qwen3 decoder tensors (embed, 28 attention/MLP layers, norms), with no EAGLE or Medusa draft-head tensors. model/config.json is a plain Qwen3ForCausalLM. This is a general-purpose base model, not a specialist drafter.
✓no continuation lookupserver.py produces drafts only through MODEL.generate (greedy, 16 tokens) over the int4-dequantized network. It loads no
server.py produces drafts only through MODEL.generate (greedy, 16 tokens) over the int4-dequantized network. It loads no data files, caches, lookup tables or stored continuations. int4_model.py loads only model/model.safetensors, and its strict key check raises on any unexpected tensor. The serving path ships no other data.
✓actual inference pathserve.sh sets offline HF flags, installs the bundled wheels with --no-index, and runs server.py. server.py imports int4_
serve.sh sets offline HF flags, installs the bundled wheels with --no-index, and runs server.py. server.py imports int4_model.load_drafter, which dequantizes model/model.safetensors into AutoModelForCausalLM.from_config with a strict key check and tied lm_head. The /draft endpoint batches requests and calls generate on that model. The trajectory shows the same loader and server run against the gptq17 weights on the leased GPU (trajectory#12, trajectory#20). The served process does not call the target and makes no network calls.
✓harness credentials used only by the harnessIn the trajectory, the participant used the provided spin/ssh compute lease to run on the H100 (trajectory#12, trajector
In the trajectory, the participant used the provided spin/ssh compute lease to run on the H100 (trajectory#12, trajectory#18) and HF_TOKEN only for allowed downloads (trajectory#3). It ran a secret scan before submitting (trajectory#55 area in steps-42-62). Searches for API_KEY, BASE_URL and OpenRouter found no model-API calls, no echoed keys and no extra harness CLI launches beyond allowed spin use. The shipped code contains no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed2h 42m into the runtook 5m 50s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 17, in <module> from f
3 gates
- ✓review prerequisites
- ✓submission size
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 17, in <module> from fastapi import FastAPI, Request File "/opt/venv/lib/python3.12/site-packages/fastapi/__init__.py", line 7, in <module> from .applications import FastAPI as FastAPI File "/opt/venv/lib/python3.12/site-packages/fastapi/applications.py", line 7, in <module> from fastapi import routing File "/opt/venv/lib/python3.12/site-packages/fastapi/routing.py", line 58, in <module> from fastapi.dependencies.utils import ( File "/opt/venv/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 52, in <module> from fastapi.background import BackgroundTasks File "/opt/venv/lib/python3.12/site-packages/fastapi/background.py", line 5, in <module> from fastapi.telemetry._api import _operation File "/opt/venv/lib/python3.12/site-packages/fastapi/telemetry/__init__.py", line 1, in <module> from ._api import TelemetryConfig as TelemetryConfig File "/opt/venv/lib/python3.12/site-packages/fastapi/telemetry/_api.py", line 9, in <module> from opentelemetry import context as otel_context ModuleNotFoundError: No module named 'opentelemetry'
- 2passed2h 49m into the runtook 2m 19s
startup 63.4 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).
7 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 63.4 s
- ✓POST /draft · {"draft": [2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✓held-out serving time (projected) · projected from 503 of about 9,165 held-out requests at concurrency 32: 22.4 s of serving over 31 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.5 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
- ✓request failures · 0 of 503 requests failed, timed out or were malformed; each scores as an empty draft
- ✓healthy after requests
The agent's last message before its session ended.
The draft server is submitted. It serves a 4-bit Qwen3-1.7B drafter for greedy speculative decoding of Qwen3-8B.
On the public examples the mean acceptance length was 4.73 (score about 0.05 above the 4.6 floor). The official self-check passed: health in 63 seconds, no failed requests, and projected held-out serving time of about 410 seconds against the 10,800-second budget. The submission is 988 MB.