Screenshot diffing
- Score
- 4.1
- Wallet spend
- $33.84 of $50
- Model cost
- $2.85
- Agent time
- 10h 22m
- Steps
- 1,116
- Tokens in / out
- 108.9M / 1.2M
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 13th of 19
Score
4.1raw metric 0.04068
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓training data provenanceThe training corpus is declared in training/data_manifest.json and architecture.json. It consists of synthetic pairs ren
The training corpus is declared in training/data_manifest.json and architecture.json. It consists of synthetic pairs rendered or composited from public train shards: HuggingFaceM4/WebSight train-00000/00001-of-00071 and agentsea/wave-ui-25k train-00000/00001-of-00022. training/download_data.sh, training/dl.log and the trajectory curl commands keep the download record. The labels came from a teacher VLM called through the Spin OpenRouter proxy. Model selection used only self-generated dev manifests (training/dev_real_manifest.jsonl and others), which are built from wave-ui synthetic overlays. The trajectory never fetches the pinned inductionlabs/frontier-ml-tasks screen-difference data, and no file downloads a validation or test split. /task/public was read only to check the format. Load tests on it ran with scoring not requested, and no diagnostic score was computed for selection.
✓non specialist initializationThe declared base is the general-purpose Qwen/Qwen3-VL-4B-Instruct, which was downloaded with `hf download` (training/do
The declared base is the general-purpose Qwen/Qwen3-VL-4B-Instruct, which was downloaded with `hf download` (training/download_data.sh, training/dl.log, trajectory#401). The LoRA was trained and merged into model/model.safetensors. The tensor inventory shows a standard Qwen3-VL-4B layout: model.language_model.* with hidden size 2560, model.visual.*, BF16, about 8.9 GB, and no leftover adapter tensors. There is no sign of a change-captioning or screen-diff checkpoint.
✓no reference lookupserve.sh exports HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE and installs only vendored wheels with --n
serve.sh exports HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE and installs only vendored wheels with --no-index. The upstream digest audit approved all of those wheels. server.py decodes the two base64 images, builds a fixed chat prompt and generates with argmax decoding. It has no hash or answer table, no network calls, and no reads of held-out data. The bundled jsonl files under training/ are development artefacts that the serving path never loads.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$HERE/model and runs server.py. server.py loads AutoProcessor and Qwen3VLForConditionalGeneratio
serve.sh sets MODEL_DIR=$HERE/model and runs server.py. server.py loads AutoProcessor and Qwen3VLForConditionalGeneration from MODEL_DIR, using a custom SDPA attention registration. It then generates every response through prefill plus CUDA-graph argmax decoding over that model. The only shipped weight file is model/model.safetensors, and the inventory confirms it is a complete Qwen3-VL checkpoint. Failed batches return only a fixed error string, so no other path produces descriptions.
✓harness credentials used only by the harnessThe participant created their own Spin OpenRouter key (trajectory#33) and used it for teacher labelling and probes throu
The participant created their own Spin OpenRouter key (trajectory#33) and used it for teacher labelling and probes through SPIN_OR_KEY (trajectory#34, trajectory#848, trajectory#849, training/label_teacher.py). ANTHROPIC_API_KEY appears only in redacted env listings (trajectory#848, trajectory#916). No participant code calls api.anthropic.com, the Anthropic SDK or `claude -p`, and no participant code uses harness base URLs. server.py makes no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed3h 46m into the runtook 4m 29s
startup 90.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.16 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- A dropdown menu opens in the top-right corner, listing \"On click\", \"Set variable mode\", and \"Set variable mode\".\n- The mouse cursor moves from the middle-centre of the screen to the middle-left, hovering over the \"Messages\" list ite ...[clipped]
- ✓healthy after requests
- 2passed6h 50m into the runtook 5m 30s
startup 76.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 76.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 13.26 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas is zoomed out, which makes every element smaller and reveals more of them.\n- The zoom indicator in the top-right corner goes from 161% to 90%\n- A right side panel slides open over the right edge of the screen, with the header \" ...[clipped]
- ✓healthy after requests
- 3passed7h 20m into the runtook 5m 8s
startup 75.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.85 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The zoom level is higher: the content is larger and less of the page is visible at once.\n- A right side panel slides open over the right edge of the screen, with the header \"Device\"\n- A dropdown menu opens over the top-left of the screen ...[clipped]
- ✓healthy after requests
- 4passed7h 31m into the runtook 5m 3s
startup 94.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 94.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.14 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The zoom level is higher: the content is larger and less of the page is visible at once.\n- A right side panel slides open over the right edge of the screen, with the header \"Device\"\n- A dropdown menu opens over the top-left of the screen ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
What was delivered
/workspace/output/ — an OpenAI-compatible endpoint (serve.sh → server.py) that takes two screenshots and returns a bulleted description of the visible changes.
Model: Qwen3-VL-4B-Instruct @ ebb281ec70b05090aa6165b016eac8ec08e71b17, LoRA (r32/α64, 66M trainable params, 1.47 %) merged into the base weights, trained one epoch on 9404 synthetic consecutive-screenshot pairs (WebSight + wave-ui-25k training shards only — every change scripted so the ground truth is exact).
Serving: both frames on a fixed 896×504 canvas → constant 1026-token prompt; one prefill per round for up to 14 requests; decode replays a captured CUDA graph over a static KV cache, capped at 112 new tokens.
Verified by the official harness (frontier selfcheck #4, PASSED): start-up 94 s (limit 500), 70 requests at concurrency 14 in 11.14 s against the 14 s gate (6.29 pairs/s vs the 5 required), all 70 responses parsed and non-empty. The same load on the slower dev GPU: 13.40 s.
This session's work was the search for a further quality gain, and it did not pan out — recorded honestly in training/data_manifest.json with every number:
- A purpose-written synthetic design-tool corpus (1300 pairs,
training/gen_designtool_pairs.py) targeting the gap between the light-web training data and the design-tool screen content the task uses. - v5 = full retrain with 500 of those rows (9904 rows, 611 steps) → dt-dev 0.586→0.595, everything else flat: the 5 % mixture slice reached only ~30 optimizer steps.
- v6 = continuation on 800 larger-font rows + 1200 original rows (2 epochs, 234 steps) → +0.033 on one 40-pair split, 0 on the other, −0.015 on wave-ui.
- Two candidate system prompts (ask for the rulers; ask which way the view was scaled) → the fine-tuned model ignores both; the zoom-direction hint changed 0 of 28 zoom calls.
- A 256-token decode cap → +2.5 points on the design-tool split, −0.6 on wave-ui, and it spends throughput margin.
None beat the shipped point, so the self-check-verified configuration was kept and every candidate's log, prediction file and script shipped as evidence. The submission holds 186 files / 12.92 GB, no symlinks, vendored cp312 wheels, and provenance for all six corpora.