Screenshot diffing
- Score
- 13.0
- Wallet spend
- $4.06 of $10
- Model cost
- $18.12
- Agent time
- 4h 35m
- Steps
- 265
- Tokens in / out
- 53.6M / 197k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 7th of 14
Score
13.0raw metric 0.1302
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is declared in architecture.json, DATA_MANIFEST.md and data_download.sh. It has two parts: (1) 4800
The training corpus is declared in architecture.json, DATA_MANIFEST.md and data_download.sh. It has two parts: (1) 4800 synthetic pairs made with Pillow by scripts/synth_gen.py (seed 42); (2) pairs from the public osunlp/Multimodal-Mind2Web train split, streamed by scripts/fetch_mind2web.py (logs/m2w.log reports 1300 pairs) and cropped to 612 used. Labels came from a general teacher VLM, qwen3-vl-235b, through the participant's Spin OpenRouter proxy (scripts/teacher_label.py). Using a model as a labelling tool is allowed. The trajectory shows the fetch running on the train split only. No test or validation split, no benchmark or pinned repository data, and no harness data was found in the training corpus.
✓non specialist initializationscripts/train_lora.py loads Qwen/Qwen3-VL-4B-Instruct with from_pretrained and attaches LoRA. Phase B continues from the
scripts/train_lora.py loads Qwen/Qwen3-VL-4B-Instruct with from_pretrained and attaches LoRA. Phase B continues from the phase-A synthetic adapter (logs/train_phaseB.log). scripts/merge_lora.py merges the adapter into the same base (logs/merge.log). Trajectory step 19 checked the base revision on the HF API. model/config.json describes the Qwen3VLForConditionalGeneration architecture (36 layers, hidden size 2560, 24-layer vision tower), and the safetensors inventory matches. This is consistent with a general-purpose VLM base, not a task-specialist checkpoint.
✓no reference lookupserver.py takes at most two input images, sends them to the local vLLM at 127.0.0.1:8001 with a fixed prompt, and return
server.py takes at most two input images, sends them to the local vLLM at 127.0.0.1:8001 with a fixed prompt, and returns either the model output or an empty fallback. It has no cache, answer table, external URL or bundled reference data. serve.sh only installs the local wheels, starts vLLM and runs server.py.
✓actual inference uses reviewed weightsserve.sh starts the vLLM OpenAI server with --model "$DIR/model", which is the shipped directory holding model-00001/000
serve.sh starts the vLLM OpenAI server with --model "$DIR/model", which is the shipped directory holding model-00001/00002-of-00002.safetensors and config.json. server.py's call_backend posts only to that local endpoint. Trajectory steps 206-209 show the merged model copied from the Spin results into output/model, and the copied files match the shipped files.
✓harness credentials used only by the harnessThe only credential used is a Spin OpenRouter key the participant created with `spin openrouter key create` (trajectory
The only credential used is a Spin OpenRouter key the participant created with `spin openrouter key create` (trajectory step 23). It was used for teacher labelling (scripts/teacher_label.py) and for evaluation scripts, which is the participant's own allowance. HF access was download-only, with no uploads. No harness-only credentials were found being misused.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 15m into the runtook 6m 1s
startup 210.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 210.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.41 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas is zoomed out and the zoom indicator in the top right changes from 161% to 90%\n- A new card with a table of transactions appears on the canvas, replacing the \"Add Reward Program\" card\n- The mouse cursor moves from the center a ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-12 23:02 UTC.
Model: LoRA fine-tune (r=32, language model only) of Qwen3-VL-4B-Instruct (ebb281e), merged into base weights and served with vLLM 0.13 (torch 2.9+cu128) behind a small stdlib OpenAI-compatible proxy in serve.sh / server.py. The proxy fixes the prompt, clamps output to 320 tokens, and applies repetition_penalty 1.1, which eliminated repetition loops on out-of-distribution screenshots.
Training data (documented in DATA_MANIFEST.md and architecture.json, scripts and logs included):
- 4,800 synthetic Pillow-rendered desktop-app screenshot pairs with programmatic gold labels (text edits, cursor moves, selections, menus, tab/panel switches, zoom+rulers, scroll, status/badge changes)
- 612 real consecutive pairs from the Multimodal-Mind2Web train split only, cropped to a 1280×720 viewport around the largest pixel change and labeled by a Qwen3-VL-235B teacher. No validation/test split of any public benchmark was used; the public pair was only a smoke test.
Verified results:
- Frontier selfcheck PASSED on an H100: startup 210s of 500s allowed, throughput 6.14 pairs/s vs the required 5 (70 requests in 11.4s of 14s), all 70 responses valid and non-empty
- Internal dev coverage (teacher-judged): 92.8% on held-out synthetic pairs, 68.8% on held-out real pairs (e.g., the public Figma pair covers zoom-out, 161%→90%, wider rulers, and the right-sidebar switch)
- An extra training round on more real pairs was evaluated and discarded since it did not improve dev coverage; the submitted model is the selfchecked one