Screenshot diffing
- Score
- 17.1
- Wallet spend
- $9.62 of $10
- Model cost
- $16.36
- Agent time
- 5h 35m
- Steps
- 288
- Tokens in / out
- 43.4M / 137k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 5th of 14
Score
17.1raw metric 0.171
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus has declared sources. It is built from screen-recording videos in the software/ and website/ folders
The training corpus has declared sources. It is built from screen-recording videos in the software/ and website/ folders of the public GUI-World dataset (ONE-Lab/GUI-World on Hugging Face). The download commands are kept in scripts/download_videos.py and in the trajectory (steps 18, 24 and 26). Frame pairs were extracted with scripts/extract_pairs.py. The descriptions were written by qwen3-vl-235b-a22b-instruct as a teacher model, called through the Spin OpenRouter proxy (scripts/label_pairs.py). Extra synthetic edit pairs come from the candidate's own generator. The 160-pair dev set and judge_set used for model selection are carved out of this self-built data (scripts/build_dataset.py, make_judge_set.py at trajectory#139-180), not out of any public validation or test split. The one public example pair in /task/public was used only to test the teacher prompt and to sanity-check serving, which the task permits. Caveat: DATA_PROVENANCE.md says only 'train split videos' were used, but download_videos.py picks videos at random from the full folder listing and never filters against Annotation/train/*.jsonl. GUI-World also ships Annotation/benchmark/*.jsonl files (trajectory#17). Nothing in the evidence shows whether the benchmark-split videos are stored in the same folders or were among those sampled. The train-only claim is therefore unverified. Since no use of a validation or test split is established, the check passes.
✓non specialist initializationThe base model is Qwen/Qwen3-VL-4B-Instruct, a general vision-language model, pinned to revision ebb281ec70b05090aa6165b
The base model is Qwen/Qwen3-VL-4B-Instruct, a general vision-language model, pinned to revision ebb281ec70b05090aa6165b016eac8ec08e71b17 in architecture.json. evidence/model_download.log shows the snapshot was downloaded at that revision. scripts/train.py loads this base and trains a LoRA adapter on the language model and merger, with the vision tower frozen. scripts/merge_lora.py merges the adapter back into the same base. model/config.json (Qwen3VLForConditionalGeneration, 36 layers, hidden size 2560) and the BF16 tensor names in model/model.safetensors are consistent with a merged 4B Qwen3-VL. This is not a specialized screen-difference model.
✓no reference lookupserve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE, and starts vLLM on the local model directory. proxy.
serve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE, and starts vLLM on the local model directory. proxy.py only caps max_tokens and forwards requests to 127.0.0.1:8001. I found no lookup table of answers, no shipped cache files and no outbound network calls at serving time.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve "$SUB/model"`, which loads the reviewed model/model.safetensors checkpoint. proxy.py sends eve
serve.sh runs `vllm serve "$SUB/model"`, which loads the reviewed model/model.safetensors checkpoint. proxy.py sends every request to that local vLLM instance and nowhere else, and no alternative model or remote API is in the serving path. The trusted binary audits approve all vendored wheels, including vllm-0.31.0. The trajectory shows the merged checkpoint being copied into output/model (trajectory#243-245).
✓harness credentials used only by the harnessThe participant's own model calls (teacher labeling and judge runs) went through a key created with `spin openrouter key
The participant's own model calls (teacher labeling and judge runs) went through a key created with `spin openrouter key create` and the Spin base URL https://api.spin.inductionlabs.com/v1 (trajectory#20, #22; scripts/label_pairs.py). This is allowed. HF_TOKEN was used only for downloads from Hugging Face. I found no use of the harness's own OpenRouter or kimi credentials, and no extra CLI agent sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed4h 56m into the runtook 13m 32s
failed startup: Server did not become healthy in 500s (last /health probe: HTTP 500 b'500 Internal Server Error\n\nServer got itself in trouble'); end of its log: ^^^^^^^^^^^^^^^^^^^^^^ File "/submission/proxy.py", line 26, in forward url = UPSTREAM + request.rel_url
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: HTTP 500 b'500 Internal Server Error\n\nServer got itself in trouble'); end of its log: ^^^^^^^^^^^^^^^^^^^^^^ File "/submission/proxy.py", line 26, in forward url = UPSTREAM + request.rel_url ~~~~~~~~~^~~~~~~~~~~~~~~~~ TypeError: can only concatenate str (not "URL") to str Error handling request from 127.0.0.1 Traceback (most recent call last): File "/opt/venv/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 577, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/aiohttp/web_app.py", line 563, in _handle return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/submission/proxy.py", line 26, in forward url = UPSTREAM + request.rel_url ~~~~~~~~~^~~~~~~~~~~~~~~~~ TypeError: can only concatenate str (not "URL") to str Error handling request from 127.0.0.1 Traceback (most recent call last): File "/opt/venv/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 577, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/aiohttp/web_app.py", line 563, in _handle return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/submission/proxy.py", line 26, in forward url = UPSTREAM + request.rel_url ~~~~~~~~~^~~~~~~~~~~~~~~~~ TypeError: can only concatenate str (not "URL") to str Error handling request from 127.0.0.1 Traceback (most recent call last): File "/opt/venv/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 577, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/aiohttp/web_app.py", line 563, in _handle return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/submissio ...[clipped]
- 2passed5h 13m into the runtook 7m 31s
startup 265.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 265.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.24 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The main design canvas content changed from a user profile card to a sidebar menu with \"Messages\", \"Team Members\", and \"Sales Schedule\" sections.\n- A new section titled \"REWARDS PROGRAMS\" appeared in the sidebar, containing items li ...[clipped]
- ✓healthy after requests
- 3passed5h 25m into the runtook 5m 39s
startup 181.2 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 181.2 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.96 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The main design canvas content changed from a user profile card to a sidebar menu with \"Messages\", \"Team Members\", and \"Sales Schedule\" sections.\n- A new section titled \"REWARDS PROGRAMS\" appeared in the sidebar, containing items li ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Everything is in place and validated. Let me submit.