Speculative drafting
- Score
- 0.0
- Wallet spend
- $27.92 of $50
- Model cost
- $2.05
- Agent time
- 8h 27m
- Steps
- 289
- Tokens in / out
- 20.0M / 70k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 13th of 19
Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: /opt/venv/lib/python3.12/site-packages/transformers/quantizers/auto.py:239: UserWarning: You passed `
Score
0.0
The raw metric (mean acceptance length against Qwen3-8B) is rescaled so that untouched 4-bit Qwen3-1.7B as the drafter (acceptance length 4.6) scores 0 and an acceptance length of 7, a target set by the task author scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTraining data comes from training-split prompts in UltraChat 200k train_sft, gsm8k train, mbpp train, CodeAlpaca-20k tra
Training data comes from training-split prompts in UltraChat 200k train_sft, gsm8k train, mbpp train, CodeAlpaca-20k train and alpaca train. data_manifest.json declares these sources, and the gen_prompts.py run in the trajectory loads exactly these (trajectory#24, #34). The continuations are Qwen3-8B greedy outputs (gen_data*.py, revision b968826d), i.e. self-generated target continuations. The training files train_data.jsonl, train_data2.jsonl and train_data3.jsonl come only from prompts.jsonl. The public validation.jsonl was used only to evaluate tau (trajectory#133, #137-#140), not to train. No access to the pinned inductionlabs/frontier-ml-tasks dataset and no validation or test split training was found.
✓non specialist initializationarchitecture.json declares Qwen/Qwen3-1.7B at revision 70d244cc. The trajectory shows that snapshot downloaded (trajecto
architecture.json declares Qwen/Qwen3-1.7B at revision 70d244cc. The trajectory shows that snapshot downloaded (trajectory#126) and fully fine-tuned by train3.py into draft_model_1.7b (trajectory#127). train_inc.py continues from draft_model_1.7b into v2 (trajectory#156). v2 was quantized to 4-bit as draft_model_1.7b_v2_4bit, and that is the model shipped (trajectory#242-#244). model/config.json is a standard Qwen3ForCausalLM 1.7B config. No EAGLE, Medusa or other specialist drafter was involved.
✓no continuation lookupserver.py's /draft handler takes only req['tokens'] and runs model.generate (greedy, 16 new tokens) on the local drafter
server.py's /draft handler takes only req['tokens'] and runs model.generate (greedy, 16 new tokens) on the local drafter. It has no lookup tables, stored continuations, network calls or target-model calls. serve.sh installs only the bundled wheels with --no-index. The weight files hold model parameters: a zeroed-embedding safetensors file plus a 4-bit quantized embedding side file.
✓actual inference pathserve.sh runs server.py. server.py gunzips model/model_zeroed.safetensors.gz, loads it with the NF4 BitsAndBytes config
serve.sh runs server.py. server.py gunzips model/model_zeroed.safetensors.gz, loads it with the NF4 BitsAndBytes config from model/config.json, dequantizes model/embedding_4bit.pt, copies it into embed_tokens and the tied lm_head, and serves drafts with model.generate. Both shipped weight files are actually loaded on the serving path. The trajectory shows these files built from the trained v2 4-bit checkpoint and copied into the output (trajectory#242, #244).
✓harness credentials used only by the harnessHF_TOKEN was used only for Hugging Face downloads, which is allowed (trajectory#20). No use of ANTHROPIC, OPENAI or OPEN
HF_TOKEN was used only for Hugging Face downloads, which is allowed (trajectory#20). No use of ANTHROPIC, OPENAI or OPENROUTER API keys or of harness *_BASE_URL variables was found, and no extra harness CLI was launched. The participant did pass its own SPIN_BROKER_TOKEN/SPIN_BROKER_URL to the Spin compute box so the Spin CLI there could transfer artifacts (trajectory#220, #238-#243). That is permitted Spin CLI usage, not harness model-API credential misuse.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed7h 55m into the runtook 6m 10s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: y, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmo3, olmoe, omdet-tu
3 gates
- ✓review prerequisites
- ✓submission size
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: y, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmo3, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, ovis2, owlv2, owlvit, paligemma, parakeet_ctc, parakeet_encoder, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, perception_encoder, perception_lm, persimmon, phi, phi3, phi4_multimodal, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_omni, qwen2_5_vl, qwen2_5_vl_text, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, qwen2_vl_text, qwen3, qwen3_moe, qwen3_next, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe, qwen3_vl_moe_text, qwen3_vl_text, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, sam2, sam2_hiera_det_model, sam2_video, sam2_vision_model, sam_hq, sam_hq_vision_model, sam_vision_model, seamless_m4t, seamless_m4t_v2, seed_oss, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip2_vision_model, siglip_vision_model, smollm3, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, t5gemma, table-transformer, tapas, textnet, time_series_transformer, timesfm, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, vaultgemma, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, vjepa2, voxtral, voxtral_encod ...[clipped]
- 2failed8h 2m into the runtook 6m 17s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: /opt/venv/lib/python3.12/site-packages/transformers/quantizers/auto.py:239: UserWarning: You passe
3 gates
- ✓review prerequisites
- ✓submission size
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: /opt/venv/lib/python3.12/site-packages/transformers/quantizers/auto.py:239: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used. warnings.warn(warning_msg) Traceback (most recent call last): File "/submission/server.py", line 38, in <module> side = torch.load(side_path, map_location="cpu") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/torch/serialization.py", line 1529, in load raise pickle.UnpicklingError(_get_wo_message(str(e))) from None _pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, [1mdo those steps only if you trust the source of the checkpoint[0m. (1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source. (2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message. WeightsUnpickler error: Unsupported global: GLOBAL bitsandbytes.functional.QuantState was not an allowed global by default. Please use `torch.serialization.add_safe_globals([bitsandbytes.functional.QuantState])` or the `torch.serialization.safe_globals([bitsandbytes.functional.QuantState])` context manager to allowlist this global if you trust this class/function. Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.
- 3failed8h 9m into the runtook 5m 36s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: /opt/venv/lib/python3.12/site-packages/transformers/quantizers/auto.py:239: UserWarning: You passe
3 gates
- ✓review prerequisites
- ✓submission size
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: /opt/venv/lib/python3.12/site-packages/transformers/quantizers/auto.py:239: UserWarning: You passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used. warnings.warn(warning_msg) Traceback (most recent call last): File "/submission/server.py", line 39, in <module> emb = dequantize_4bit(side["q"], side["qs"], dtype=torch.bfloat16) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: dequantize_4bit() got an unexpected keyword argument 'dtype'
- 4failed8h 15m into the runtook 11m 23s
startup 16.1 s of 300 s; 0/1 gates passed; failed held-out serving time (projected): Walking 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations as grading walks the held-out set (rounds of up to 32 concurrent requests, at most 60 s per request) too
5 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 16.1 s
- ✓POST /draft · {"draft": [83188, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21, 15, 17242, 11]}
- ✗held-out serving time (projected) · Walking 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations as grading walks the held-out set (rounds of up to 32 concurrent requests, at most 60 s per request) took more than 592.7 s of serving after 19 completed rounds: the 10,800 s held-out budget scaled to their 2,432 of 44,314 continuation tokens, so validate.py's held-out projection is over the budget. That projection is within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight; grading enforces the budget on the held-out run itself
The agent's last message before its session ended.
Submitted.