Screenshot diffing
- Score
- 6.5
- Wallet spend
- $42.06 of $200
- Model cost
- $25.84
- Agent time
- 8h 34m
- Steps
- 363
- Tokens in / out
- 75.3M / 260k
- Self-checks
- 0 passed of 4 used
- Rank at $200 wallet
- 10th of 14
Score
6.5raw metric 0.06536
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data has a declared source: public Multimodal-Mind2Web train-split parquet shards, plus synthetic Playwrigh
The training data has a declared source: public Multimodal-Mind2Web train-split parquet shards, plus synthetic Playwright screenshot pairs made by code/gen_data.py. These sources are named in architecture.json and data/DATA_MANIFEST.json. The trajectory shows that only files starting with data/train- were downloaded (trajectory#135, trajectory#141). Test shards were listed at one point but never fetched. Gemini-2.5-flash teacher labels were produced on those pairs by code/label_pairs.py. Model selection used a held-out slice of the train split, chosen by md5(annotation_id) in code/make_v3_data.py. No benchmark reference data or test data appears anywhere.
✓non specialist initializationcode/train_sft.py loads Qwen2_5_VLForConditionalGeneration.from_pretrained with the default 'Qwen/Qwen2.5-VL-3B-Instruct
code/train_sft.py loads Qwen2_5_VLForConditionalGeneration.from_pretrained with the default 'Qwen/Qwen2.5-VL-3B-Instruct', a general-purpose instruct model, and fine-tunes it with the vision tower frozen. model/config.json and the safetensors tensor inventories match the standard Qwen2.5-VL-3B architecture. The shipped weights came from the v3 run of that script and were copied from /tmp/model-v3 to output/model (trajectory#340, trajectory#341). No specialist checkpoint was used.
✓no reference lookupserve.sh starts a vLLM server with HF_HUB_OFFLINE=1 and then runs code/serve_proxy.py. The proxy only forwards requests
serve.sh starts a vLLM server with HF_HUB_OFFLINE=1 and then runs code/serve_proxy.py. The proxy only forwards requests to the local vLLM replica and caps max_tokens. It contains no lookup tables, retrieval step, external API calls or bundled reference answers.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server --model "$DIR/model", which is the directory holding the inventoried mo
serve.sh runs vllm.entrypoints.openai.api_server --model "$DIR/model", which is the directory holding the inventoried model-00001/00002-of-00002.safetensors. code/serve_proxy.py routes every request to that local server, and there is no alternate model path. The trajectory shows these weights are the v3 checkpoint copied into output/model (trajectory#341).
✓harness credentials used only by the harnessThe teacher-labeling calls in code/label_pairs.py used OR_BASE_URL and OR_KEY. That key came from the participant's own
The teacher-labeling calls in code/label_pairs.py used OR_BASE_URL and OR_KEY. That key came from the participant's own `spin openrouter key create` (trajectory#128), not from harness credentials. The trajectory shows no use of harness ANTHROPIC or OPENROUTER variables and no harness CLIs were launched. HF downloads used read-only access, which the policy permits.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 3m into the runtook 8m 56s
startup 194.1 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.64 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 194.1 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.64 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right sidebar changes from showing the \"Prototype\" panel to showing a \"Design system\" panel with options like \"Variant interactions\" and \"Scroll behavior\".\n- A different layer in the left layers panel is selected, and the canvas ...[clipped]
- ✓healthy after requests
- 2failed5h 20m into the runtook 8m 23s
startup 330.5 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 20.39 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 330.5 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 20.39 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right sidebar has changed from showing \"Interactions\" to showing \"Design system\".\n- The canvas area has changed from displaying a \"Add Reward Program\" dialog with a list of users to displaying a \"Messages\" section with various d ...[clipped]
- ✓healthy after requests
- 3failed5h 38m into the runtook 6m 48s
startup 223.2 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 21.37 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 223.2 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 21.37 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right sidebar's \"Interactions\" panel is no longer visible.\n- The right sidebar's \"Scroll behavior\" panel is no longer visible.\n- The right sidebar's \"Design system\" panel is no longer visible.\n- The right sidebar's \"Prototype\" ...[clipped]
- ✓healthy after requests
- 4failed8h 5m into the runtook 7m 33s
startup 215.4 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 27.57 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 215.4 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 27.57 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right sidebar's \"Interactions\" panel is no longer open.\n- The right sidebar's \"Scroll behavior\" panel is no longer open.\n- The right sidebar's \"Design system\" panel is no longer open.\n- The right sidebar's \"Prototype\" panel is ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of the delivered solution in /workspace/output/ (191 files, 12.5 GB):
Model — Qwen2.5-VL-3B-Instruct (66285546d2b821cf421d4f5eb2576359d3770cd3) full-SFT of the language model (vision tower frozen), 1 epoch on 12,538 pairs:
- 8,000 synthetic 1280×720 screenshot pairs rendered in headless Chromium across 6 app templates (dashboard, design tool, spreadsheet, email, editor, chat) with recorded edits — text/number changes, cursor moves, selections, menus, panel swaps, zoom+ruler effects, status changes (gen_data.py)
- 2,269 real pairs from the Multimodal-Mind2Web train split only (10 train shards; consecutive trajectory steps cropped to 1280×720 around the acted element), labeled by Gemini 2.5 Flash via Spin OpenRouter and filtered to ≤6-bullet concise targets, upsampled ×2 (extract_m2w.py, label_pairs.py, make_v3_data.py)
Selection used only held-out train-split data (never trained on): word-recall v1 synthetic-only 0.255 → v2 0.592 @145 words → v3 0.504 @104 words; synthetic held-out 0.887. No validation/test split of any public benchmark was used for training or selection; provenance and logs are in architecture.json, DATA_MANIFEST.json, and logs/.
Serving — serve.sh installs 157 vendored cp312/cu128 wheels offline, starts one vLLM 0.10.2 replica (512k-pixel image cap) behind a thin proxy (serve_proxy.py) that caps generation at 160 tokens, with sequential health + warmup probes; startup passed in all four selfchecks (~215s of 500s).
Verified risks: throughput on selfcheck hosts failed the 14s gate in all four checks (17.6–27.6s) with high host-to-host variance (warm engine measured at 1950 tok/s locally; identical stack did 70 requests in 9.2s on a rented H100). The final config minimizes every per-request cost (concise-output model, token/pixel caps, single replica — dual replicas and async scheduling were tested and rejected: async crashes the vLLM worker with this model, dual contends on 8 CPUs). Quality was traded deliberately toward passing the throughput gate, since a gate failure scores zero.