Screenshot diffing
- Score
- 35.2
- Wallet spend
- $121.33 of $200
- Model cost
- $33.97
- Agent time
- 5h 48m
- Steps
- 582
- Tokens in / out
- 56.0M / 503k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 4th of 14
Score
35.2raw metric 0.352
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.json lists where the training data came from: ONE-Lab/GUI-World train annotations, with the benchmark anno
data_manifest.json lists where the training data came from: ONE-Lab/GUI-World train annotations, with the benchmark annotations left out, and the cua-lite/OpenCUA desktop/use/train shards. The fetch scripts keep those filters: fetch_gw.py skips Annotation/benchmark and fetch_oc.py keeps only paths containing /train/. Fetch logs were kept with them. Training pairs were made by build_pkg.py from those gw/oc pools. Teacher labels came through the harness-provided spin openrouter key. The dev set used only train-pool pairs plus the public example, which is allowed. No hidden or test data source turned up.
✓non specialist initializationarchitecture.json declares the general model Qwen/Qwen3-VL-8B-Instruct, pinned to revision 0c351dd0. The trajectory show
architecture.json declares the general model Qwen/Qwen3-VL-8B-Instruct, pinned to revision 0c351dd0. The trajectory shows snapshot_download of that revision. train_sft.py loads it with from_pretrained, runs LoRA SFT on the language model, then merges the adapter. The shipped checkpoint has the standard Qwen3-VL tensor layout and config (qwen3_vl with fp8 quantization). Nothing shows a starting point that was already specialized for screen-difference.
✓no reference lookupserve.sh starts the stock vLLM OpenAI server on the local model. The server runs offline (HF_HUB_OFFLINE=1). The only ad
serve.sh starts the stock vLLM OpenAI server on the local model. The server runs offline (HF_HUB_OFFLINE=1). The only add-ons are the latency_governor middleware, which just lowers max_tokens, and gate.py, which does a synthetic warm-up and forwards bytes. Neither file contains a reference-answer table, retrieval step, or external API call. The chat template is the standard one.
✓actual inference uses reviewed weightsserve.sh points vLLM --model at $HERE/model. That directory holds config.json, the safetensors index and four FP8/BF16 s
serve.sh points vLLM --model at $HERE/model. That directory holds config.json, the safetensors index and four FP8/BF16 safetensors shards, which match the reviewed Qwen3-VL tensor inventory. The trajectory (step 452) shows the run2 fp8t checkpoint being copied into output/model. No other model or remote endpoint is used at serving time.
✓harness credentials used only by the harnessTeacher annotation in llm.py used a spin openrouter key that the participant created and saved to .or_key.json (trajecto
Teacher annotation in llm.py used a spin openrouter key that the participant created and saved to .or_key.json (trajectory steps 31-32). That is the permitted credential path. ANTHROPIC appears only in a rule statement and a final leak grep. No external claude CLI session, direct api.anthropic.com call, or reuse of the harness's own credentials was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed3h 23m into the runtook 8m 19s
startup 239.6 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 239.6 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.91 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zooms out from 161% to 90%, revealing more of the left sidebar and the full \u201cRewards\u201d panel with its three transactions.\n- The \u201cAdd Reward Program\u201d card and \u201cSam / Owner\u201d row are replaced by a taller ...[clipped]
- ✓healthy after requests
- 2passed4h 27m into the runtook 7m 34s
startup 231.2 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 231.2 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.76 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zooms out from 161% to 90%, revealing the full left sidebar and the \u201cRewards\u201d panel with \u201cMessages 5\u201d, \u201cTeam Members\u201d, \u201cSales Schedule\u201d, \u201cStamps Cards\u201d, \u201cAdd Reward Program\u2 ...[clipped]
- ✓healthy after requests
- 3passed5h 40m into the runtook 8m 24s
startup 257.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 257.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.40 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zooms out from 161% to 90%, revealing the full left sidebar and the \u201cRewards\u201d panel with \u201cMessages 5\u201d, \u201cTeam Members\u201d, \u201cSales Schedule\u201d, \u201cStamps Cards\u201d, \u201cAdd Reward Program\u2 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. The last self-check passed on the exact package I submitted (266 files, 15.8 GB), and its measured throughput is the tightest of the gates.
- Throughput: 70 requests at concurrency 14 took 11.40 s against the 14 s limit (6.1 pairs/s against the 5 required). That leaves about 2.6 s of margin, and the earlier self-check run measured 10.76 s, so the grader's timing varies by roughly half a second between runs.
- Startup:
/healthanswered after 257 s, against the 500 s limit. - Responses: all 70 parsed with a non-empty description. The check ran on the one public example pair; no judge scored the descriptions and no architecture review ran, so I have no quality score from the grader.
The model is Qwen3-VL-8B-Instruct with a LoRA trained on 4,728 teacher-written descriptions, merged and served in FP8 with vLLM 0.19.1. Output is capped at 320 tokens. A small middleware shortens answers only when a batch is running late (default deadline 12.5 s). I tested it on my own H100 box with a deliberately tight deadline of 6.5 s: batches finished in 6.2–7.0 s and the clamp did cut output length. The self-check log tail showed no clamp lines, so I can't tell whether it ever bound on the grader.
A second training run that added vision-tower training gave no gain on my proxy scores, so I shipped the first run. architecture.json and data_manifest.json document this, along with the held-out pair sets I drew from the training pool for comparing checkpoints. I never trained on those sets or on any public validation or test split.