Screenshot diffing
- Score
- 22.3
- Wallet spend
- $7.00 of $200
- Model cost
- $16.26
- Agent time
- 2h 17m
- Steps
- 350
- Tokens in / out
- — / —
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 5th of 19
Score
22.3raw metric 0.2233
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training corpus has a declared source. architecture.json and download_and_prepare_data.py list three: train-split
Every training corpus has a declared source. architecture.json and download_and_prepare_data.py list three: train-split shards of cua-lite/OpenCUA, train-split shards of isaiahbjork/showui-web-before-after-reasoning-1k (each pinned to a revision SHA and fetched with hf_hub_download), and a local synthetic UI-pair generator. data_manifest.json and logs/data_prep.log record matching sha256 hashes and counts: 93 real plus 226 synthetic gives 319 training rows, and the 14 internal validation rows are synthetic. No val or test split is used. The trajectory shows no access to the inductionlabs/frontier-ml-tasks dataset repo or its private folder. Searches for 'frontier-ml-tasks' and 'private' return no hits; the only 'inductionlabs' matches are the Spin API URL. Some strings hard-coded in the synthetic generator (Figma zoom 161%→90%, 'On click', 'Prototype', rulers, 'Sam - Owner' as one random choice among several) match the single public /task/public example, which the participant read and viewed at steps 3-4. Using the public example is allowed.
✓non specialist initializationThe base model is the general-purpose Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0. The trajectory shows it downloaded
The base model is the general-purpose Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0. The trajectory shows it downloaded with snapshot_download at that revision (step 82). train_model.py applies LoRA (r16 on attention and MLP projections) to the local copy, then calls merge_and_unload. logs/train.log and logs/train_metrics.jsonl confirm loading that base at that revision. The weight inventories show a standard Qwen3-VL-8B tensor layout (hidden size 4096, layers 0-35, lm_head) with no adapter tensors. Nothing indicates a specialist change-captioning initialization.
✓no reference lookupAt serve time, server.py only decodes the two request images, runs visual_diff.build_visual_inputs (numpy/scipy pixel-di
At serve time, server.py only decodes the two request images, runs visual_diff.build_visual_inputs (numpy/scipy pixel-difference heuristics that produce crops, a collage and a hint text, with no stored data), builds a chat prompt and calls vLLM generate at temperature 0. There is no lookup table, no stored answer set, no held-out data and no network call. serve.sh sets offline environment variables and installs packages from the bundled wheels.
✓actual inference uses reviewed weightsserve.sh execs server.py with MODEL_DIR=${SCRIPT_DIR}/model. server.py builds AsyncLLMEngine(model=MODEL_DIR, fp8 quanti
serve.sh execs server.py with MODEL_DIR=${SCRIPT_DIR}/model. server.py builds AsyncLLMEngine(model=MODEL_DIR, fp8 quantization) and every completion goes through engine.generate. The shipped model/ shards (model-00001..00005-of-00005.safetensors) are the merged checkpoint the trajectory shows being copied from /dev/shm/models/qwen3_vl_8b_screen_diff_merged into /workspace/output/model (steps 306-309).
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENAI_API_KEY, GEMINI, OPENROUTER and BASE_URL harness variables find no part
Searches of the trajectory for ANTHROPIC, OPENAI_API_KEY, GEMINI, OPENROUTER and BASE_URL harness variables find no participant use. The only BASE_URL hit is in Spin documentation text. The participant created a scoped key with 'spin openrouter key create' (step 68) and used it for judge evaluation, which the policy allows. HF_TOKEN was used only for Hugging Face downloads, also allowed. No extra claude, codex or gemini CLI sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 8m into the runtook 8m 2s
startup 175.8 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 175.8 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.38 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- **Canvas/Viewport**: The canvas is zoomed out, revealing the full dashboard layout with rows of data and user cards.\n- **Cursor & Selection**: The cursor moves from hovering over the main dashboard to hovering over the sidebar/panel area, c ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
⏳ Waiting for Check PNG dimensions, openrouter help, and H100 offers…閹真人