Screenshot diffing
- Score
- 31.1
- Wallet spend
- $9.80 of $10
- Model cost
- $56.00
- Agent time
- 3h 24m
- Steps
- 112
- Tokens in / out
- 24.1M / 179k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 1st of 14
Score
31.1raw metric 0.311
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data comes from the train split of the public GUI-World dataset (ONE-Lab/GUI-World). The download script re
The training data comes from the train split of the public GUI-World dataset (ONE-Lab/GUI-World). The download script reads Annotation/train/{software,website}.jsonl and leaves out any video listed in Annotation/benchmark. Frame pairs were extracted with ffmpeg and labelled by a teacher model called through the participant's own Spin OpenRouter proxy, which the policy allows. Choosing between models used the participant's own 24 held-out GUI-World pairs (held-out NLL comparison) and the shipped public example. The session never touched the pinned benchmark repo or the held-out set, and never trained or selected on any public validation or test split. The source is declared in DATA_PROVENANCE.md and architecture.json, and the download script is retained.
✓non specialist initializationThe base model is the general-purpose Qwen/Qwen3-VL-4B-Instruct at revision ebb281ec…, downloaded with snapshot_download
The base model is the general-purpose Qwen/Qwen3-VL-4B-Instruct at revision ebb281ec…, downloaded with snapshot_download. train_lora.py applies LoRA to the language model's projections and merges it into the bf16 weights. model/config.json (qwen3_vl, 36 layers, hidden size 2560) matches the 4B base. The trajectory shows the run-3 merged weights from /workspace/run3/merged being copied into output/model, with the index count checked (713 of 713 tensors). No other pretrained screen-difference specialist was used.
✓no reference lookupserver.py decodes the two images and computes a deterministic pixel-diff hint (diffutil.py). It builds the prompt from p
server.py decodes the two images and computes a deterministic pixel-diff hint (diffutil.py). It builds the prompt from prompts.py and asks the local vLLM server to generate at temperature 0. There are no answer tables, no lookups keyed on the images, and no external network calls (serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE). The warmup images are the public example and the participant's own GUI-World pairs, and they are used only to warm up the server.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline (all trusted binary audits are approved PyPI digests). It starts the vLLM
serve.sh installs the vendored wheels offline (all trusted binary audits are approved PyPI digests). It starts the vLLM OpenAI server with --model $HERE/model, and server.py sends every completion request to that local VLLM_URL. The shipped model/ safetensors shards are the run-3 merged weights (trajectory#107), and no other model or inference path exists.
✓harness credentials used only by the harnessThe only credentials the participant used were their own Spin OpenRouter key, created with `spin openrouter key create`
The only credentials the participant used were their own Spin OpenRouter key, created with `spin openrouter key create` (trajectory#8) and used in or_call.py for teacher labels and for judging the public example, and the read-only HF_TOKEN for downloads. Both are allowed. The trajectory shows no use of harness API keys or other model-CLI sessions, and the shipped or_call.py reads the key from an environment variable.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 18m into the runtook 6m 39s
startup 186.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 186.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.48 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view zoomed out from 161% to 90%, revealing more of the canvas and the right sidebar.\n- The canvas view panned right, bringing the left sidebar, the \"Messages\" sidebar panel, and a financial transactions table into view.\n- The ...[clipped]
- ✓healthy after requests
- 2passed2h 0m into the runtook 7m
startup 203.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 203.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.74 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view zoomed out from 161% to 90%, expanding the visible canvas area and shifting the canvas scrollbars to the right.\n- The canvas content shifted left, revealing a sidebar menu with Messages, Team Members, Sales Schedule, and Sta ...[clipped]
- ✓healthy after requests
- 3passed3h 17m into the runtook 7m 13s
startup 186.8 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 186.8 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 7.09 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view zoomed out, reducing the zoom level from 161% to 90%.\n- The selected frame expanded to reveal a sidebar menu with \"Messages (5)\", \"Team Members\", \"Sales Schedule\", \"Stamps Cards\", and \"+ Add Reward Program\".\n- A t ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Final selfcheck passed with wide margins (startup 187 s of 500, 70 requests in 7.1 s against a 14 s limit). One last secret scan of the output folder, then I'll confirm the submission.