Screenshot diffing
- Score
- 34.4
- Wallet spend
- $41.69 of $50
- Model cost
- $57.48
- Agent time
- 7h 8m
- Steps
- 296
- Tokens in / out
- 79.6M / 295k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 1st of 14
Score
34.4raw metric 0.3445
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training source is declared in architecture.json and DATA_MANIFEST.md. The first is the public ONE-Lab/GUI-World A
Every training source is declared in architecture.json and DATA_MANIFEST.md. The first is the public ONE-Lab/GUI-World Annotation/train split, at a pinned revision and downloaded in three batches. The scripts and logs for those downloads are retained. The second is the public gui-wm/transition_data_v1 train split, also pinned, with an extraction log. Teacher labels came from openai/gpt-6-luna through the spin OpenRouter broker. The labels are retained in teacher_labels*.jsonl.gz, along with the labeling logs. The GUI-World benchmark annotation list was downloaded only so its videos could be excluded (trajectory#42; overlap 0). The download scripts filter out benchmark_videos.json and the dev videos. Before shipping, a leak check at step 288 reported 0 dev leaks in train_v3. The final training mix is built by build_train_v2_v3.py, and it excludes the dev videos. Dev splits come from GUI-World train videos. A search found no reference to the forbidden repo inductionlabs/frontier-ml-tasks, and no evidence that any task validation or test split was used for training. The single public example was used only for sanity and teacher comparisons (trajectory#28).
✓non specialist initializationtrain_sft.py loads its weights with AutoModelForImageTextToText.from_pretrained(--base). The base directory /root/models
train_sft.py loads its weights with AutoModelForImageTextToText.from_pretrained(--base). The base directory /root/models/Qwen3.5-4B was filled by 'hf download Qwen/Qwen3.5-4B' (trajectory steps 35-54). That is a general-purpose base model, not a screen-difference specialist. v3_train_log.txt shows base=/root/models/Qwen3.5-4B, class Qwen3_5ForConditionalGeneration, 3570M trainable parameters, and full fine-tuning on train_v3.jsonl. write_checkpoint replaces language-model tensors in the base checkpoint's own layout ('426 tensors replaced, 312 kept from base').
✓no reference lookupserve.py sends both prepped images and a fixed prompt to the vLLM AsyncLLM. Decoding is greedy, with a stop string, a re
serve.py sends both prepped images and a fixed prompt to the vLLM AsyncLLM. Decoding is greedy, with a stop string, a repetition cut and a deadline cut. The only canned text is an error fallback. There are no answer tables, no hashing of inputs against stored outputs, and no network calls at serve time. serve.sh also sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE. prompting.py only resizes images and builds the prompt. Its optional pixel-diff hint is computed from the input images and is off by default.
✓actual inference uses reviewed weightsserve.sh installs vendored wheels and execs serve.py. serve.py builds AsyncLLM with model=MODEL_DIR, which defaults to .
serve.sh installs vendored wheels and execs serve.py. serve.py builds AsyncLLM with model=MODEL_DIR, which defaults to ./model, with Qwen3.5 MTP speculative decoding. The model/ directory holds the reviewed safetensors shards: model.safetensors-00001-of-00002.safetensors contains model.language_model.* BF16 tensors, plus shard 00002. In trajectory#287 the v3 checkpoint produced by train_sft.py (logged in v3_train_log.txt) was copied into /workspace/output/model, and the shard integrity check passed (738 index keys). No other model is loaded and no remote API is called during serving.
✓harness credentials used only by the harnessThe participant obtained teacher and judge model access only through their own scoped key from 'spin openrouter key crea
The participant obtained teacher and judge model access only through their own scoped key from 'spin openrouter key create' (trajectory#26). That key is stored in /tmp/or_key.json and read by orclient.py, which is allowed. HF_TOKEN was used only for HF downloads. Searches found no use of ANTHROPIC_API_KEY or other harness credentials in the shipped code or the session, and no launched claude, codex or similar sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 50m into the runtook 7m 47s
startup 250.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 250.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.12 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The center canvas is zoomed out: the view changes from 161% to 90%, so more of the page is visible and the reward-program card and Sam/Alex card appear smaller.\n- The canvas content shifts left and slightly upward with the zoom change. The ...[clipped]
- ✓healthy after requests
- 2passed4h 23m into the runtook 7m 2s
startup 109.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 109.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 13.56 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zoom changes from 161% to 90%, shown in the top-right zoom control.\n- The canvas content becomes smaller and shifts left: the \"Add Reward Program\" card and the Sam/Alex card move left, and more of the underlying page is visible ...[clipped]
- ✓healthy after requests
- 3passed5h 11m into the runtook 5m 37s
startup 103.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 103.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.26 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zoom changes from 161% to 90%, shown in the top-right zoom control.\n- The canvas content becomes smaller and shifts left: the \"Add Reward Program\" card and the Sam/Alex card move left, and more of the underlying page is visible ...[clipped]
- ✓healthy after requests
- 4passed7h 1m into the runtook 7m
startup 103.3 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 103.3 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.99 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas is zoomed out: the top-right zoom indicator changes from \"161%\" to \"90%\".\n- The center canvas now shows a wider view of the system. The upper-left \"Add Reward Program\" area and the Sam card are smaller and positioned differ ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission went in at 12:13:56 UTC and will now be graded as it stands.