Screenshot diffing
- Score
- 3.4
- Wallet spend
- $9.00 of $50
- Model cost
- $10.79
- Agent time
- 5h 56m
- Steps
- 267
- Tokens in / out
- 53.1M / 251k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 10th of 14
Score
3.4raw metric 0.0344
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data is declared in data_manifest.md and architecture.json. code/gen_data.py streams HuggingFaceM4/WebSight
The training data is declared in data_manifest.md and architecture.json. code/gen_data.py streams HuggingFaceM4/WebSight with split="train" and applies synthetic PIL edits with templated labels. Both the 6000 train pairs and the 200 valid pairs are carved from WebSight train, and logs/gen.log confirms those counts. The manifest says no calibration or selection corpus was used. In the trajectory, other datasets (Mind2Web train keys, and HF API searches for aitw, android-control and os-atlas) were only probed, never used. Searches found no download of inductionlabs/frontier-ml-tasks. The public pair was used only for eyeball checks. The round-2 generator was a copy of gen_data.py and still used WebSight train, and round 2 was abandoned anyway. The download is not retained as a separate log beyond gen.log, but the source is a declared public train split.
✓non specialist initializationThe base is the general VLM Qwen/Qwen2.5-VL-3B-Instruct at a pinned revision, fetched by snapshot_download into /workspa
The base is the general VLM Qwen/Qwen2.5-VL-3B-Instruct at a pinned revision, fetched by snapshot_download into /workspace/base. code/train.py loads it and applies LoRA (r=32). code/merge.py merges the adapter into the base. The shipped model/model.safetensors has the Qwen2.5-VL-3B tensor structure (36 layers, hidden 2048, 32-layer vision tower) with no extra tensors. The trajectory shows round-2 continued from the round-1 LoRA but was not shipped; the final message and the output timestamps confirm the round-1 weights were submitted.
✓no reference lookupserve.sh only installs the bundled wheels offline (HF_HUB_OFFLINE=1), launches the stock vLLM OpenAI api_server on $ROOT
serve.sh only installs the bundled wheels offline (HF_HUB_OFFLINE=1), launches the stock vLLM OpenAI api_server on $ROOT/model and runs warmup.py against localhost. There is no custom handler, cached answer table, retrieval step or remote call. The model answers from its weights.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server with --model "$ROOT/model". That directory holds the reviewed model/mod
serve.sh runs vllm.entrypoints.openai.api_server with --model "$ROOT/model". That directory holds the reviewed model/model.safetensors (7,509,338,080 bytes, BF16, Qwen2.5-VL structure) and model/config.json. No alternative weight path, adapter swap or network model is configured. The bundled wheels are approved by the trusted binary audits.
✓harness credentials used only by the harnessThe trajectory shows only the participant's own use of spin compute, R2 session credentials for moving checkpoints, and
The trajectory shows only the participant's own use of spin compute, R2 session credentials for moving checkpoints, and the HF read token for downloads. The only OpenRouter interaction was `spin openrouter --help`, which goes through the allowed Spin path. No participant code reads or uses the harness API key or base URL, and no extra harness CLI was launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed3h 55m into the runtook 12m 45s
failed startup: Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: del_name_or_path, *inputs, **kwargs) [0;36m(APIServer pid=198)[0;0m ^^^^^^^^^^^^^^^^^
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: del_name_or_path, *inputs, **kwargs) [0;36m(APIServer pid=198)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0;36m(APIServer pid=198)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained [0;36m(APIServer pid=198)[0;0m return cls._from_pretrained( [0;36m(APIServer pid=198)[0;0m ^^^^^^^^^^^^^^^^^^^^^ [0;36m(APIServer pid=198)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained [0;36m(APIServer pid=198)[0;0m tokenizer = cls(*init_inputs, **init_kwargs) [0;36m(APIServer pid=198)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0;36m(APIServer pid=198)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 120, in __init__ [0;36m(APIServer pid=198)[0;0m super().__init__( [0;36m(APIServer pid=198)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py", line 178, in __init__ [0;36m(APIServer pid=198)[0;0m super().__init__(**kwargs) [0;36m(APIServer pid=198)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1472, in __init__ [0;36m(APIServer pid=198)[0;0m self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens) [0;36m(APIServer pid=198)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens [0;36m(APIServer pid=198)[0;0m self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys()) [0;36m(APIServer pid=198)[0;0m ...[clipped]
- 2passed4h 8m into the runtook 6m 19s
startup 210.3 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 210.3 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.15 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The cursor moved from in the center to near the bottom edge.\n- A context menu appeared in the top-right showing \"On click\" and \"Set variable mode\".\n- The mouse pointer became visible in the center.\n- The view scrolled down: content mo ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. LoRA fine-tune of Qwen2.5-VL-3B-Instruct on 6,000 synthetic screen-difference pairs (cursor, text, selection, menu, panel, zoom, scroll, status edits over WebSight train screenshots), served via vLLM 0.12 with vendored cu128 wheels.
Selfcheck #2 on H100 passed: startup 210s/500s, 8.6 pairs/s (gate 5), 70/70 non-empty descriptions. A round-2 continued fine-tune scored perfectly on synthetic but regressed on the real public pair, so the round-1 weights were submitted.