Screenshot diffing
- Score
- 38.0
- Wallet spend
- $172.35 of $200
- Model cost
- $69.50
- Agent time
- 6h 19m
- Steps
- 354
- Tokens in / out
- 123.4M / 329k
- Self-checks
- 3 passed of 4 used
- Rank at $200 wallet
- 3rd of 14
Score
38.0raw metric 0.3797
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training, dev and selection source is declared and public: - GUI-World train split at revision 7901fa3f… with Anno
Every training, dev and selection source is declared and public: - GUI-World train split at revision 7901fa3f… with Annotation/benchmark videos excluded in code. - VideoCUA raw_data at revision 6ecc530d…. Both are declared in architecture.json and training/DATA_MANIFEST.md, and the download scripts pin those revisions. The session confirms that the batch B and dev2 selections use train-split annotations minus benchmark videos. Dev sets come from held-out train-split videos. Teacher labels came through the Spin OpenRouter teacher route. No use of the forbidden task dataset repo and no validation/test splits were found. Public example media was used only for serving smoke tests.
✓non specialist initializationThe base is Qwen/Qwen3.5-4B, a general VLM, downloaded at pinned revision 851bf6e…. The final run C was launched as `tra
The base is Qwen/Qwen3.5-4B, a general VLM, downloaded at pinned revision 851bf6e…. The final run C was launched as `train_sft.py --model /root/models/Qwen3.5-4B` (trajectory#271-295). train_sft.py applies a PEFT LoRA to the language layers, keeps the vision tower frozen, then merges the LoRA back into the base shards. The shipped tensor inventory matches the Qwen3.5-4B layout (embedding [248320, 2560]). The model shipped in /submission/model is checkpoint C, copied from s3 modelC.
✓no reference lookupserve/proxy.py only forwards requests to the local vLLM server at 127.0.0.1. It adds a hint computed from the pixel diff
serve/proxy.py only forwards requests to the local vLLM server at 127.0.0.1. It adds a hint computed from the pixel difference by serve/diffhint.py (cv2/numpy) and cleans up the output. Its cache stores only those computed hints, keyed by a hash of the image URLs. It contains no table of descriptions or answers. serve.sh runs with HF_HUB_OFFLINE=1, and there are no external network calls.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve "$HERE/model"` locally (speculative decoding uses the model's own MTP head), then exec's serve
serve.sh runs `vllm serve "$HERE/model"` locally (speculative decoding uses the model's own MTP head), then exec's serve/proxy.py, which forwards every request to that server. There is no alternate model or remote API in the serving path. The model/ shards are the reviewed safetensors files.
✓harness credentials used only by the harnessThe participant used only their own Spin OpenRouter key, created with `spin openrouter key create` and read by training/
The participant used only their own Spin OpenRouter key, created with `spin openrouter key create` and read by training/orclient.py. HF_TOKEN was used only for Hugging Face downloads. Searches of the session found no use or copying of harness-only credentials, such as an Anthropic key or harness base URLs, and no `claude -p` calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed3h 30m into the runtook 6m 45s
startup 160.8 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.28 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 160.8 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.28 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zoomed out from a close-up of the \"Add Reward Program\" popover and the Sam/Alex popover to a wider view of the whole page.\n- The zoom level in the top-right toolbar changed from 161% to 90%.\n- The \"On click\" / \"Set variable ...[clipped]
- ✓healthy after requests
- 2passed4h 25m into the runtook 6m 11s
startup 132.3 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 132.3 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 13.77 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas scrolled horizontally to the left, so the \"Add Reward Program\" popover moved from the top-left of the canvas to the left edge, and the Sam/Alex popover moved left with it.\n- The popover's \"Add Reward Program\" title is now hid ...[clipped]
- ✓healthy after requests
- 3passed4h 44m into the runtook 5m 32s
startup 130.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 130.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.35 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas scrolled horizontally to the left, so the \"Add Reward Program\" popover moved from the top-left of the canvas to the left edge, and the Sam/Alex popover moved left with it.\n- The popover's \"Add Reward Program\" title is now hid ...[clipped]
- ✓healthy after requests
- 4passed6h 6m into the runtook 9m 15s
startup 157.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 157.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.69 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- Canvas zoomed out: \"Add Reward Program\" card and Sam/Alex popover appear smaller; \"Sam / Owner\" card fully visible.\n- Right panel switched from Prototype (Interactions, Scroll behavior) to Design (Device, Variants, Flows).\n- Zoom level ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Everything checks out. Submitting.