Screenshot diffing
- Score
- 10.6
- Wallet spend
- $31.41 of $50
- Model cost
- $1.55
- Agent time
- 7h 53m
- Steps
- 548
- Tokens in / out
- 50.3M / 447k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 7th of 19
Score
10.6raw metric 0.1056
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓training data provenancedata_manifest.json and training/data_download.sh declare two training sources. The first is the WUICC train split only (
data_manifest.json and training/data_download.sh declare two training sources. The first is the WUICC train split only (zhangsan11111122/WUICC train.zip, pinned at revision 6a781b51..., 3,000 changed and 200 no-change pairs). The second is 9,600 synthetic pairs the participant generated with training/appgen.py. training/build_data.py reads only the WUICC train directory and the synthetic directory. The trajectory confirms the same: steps 63-65 list the repo and check file sizes, step 66 downloads only train.zip, and the build command at steps 194-217 matches the manifest. No val or test zip was downloaded. There is no access to inductionlabs/frontier-ml-tasks screen-difference; 'inductionlabs' appears only in Spin/OpenRouter key setup, and 'screen-difference' only in the task prompt and the public validation example. Model selection used training loss only. Minor caveat: step 66 downloaded train.zip via resolve/main rather than the pinned revision. This does not change what was trained on.
✓non specialist initializationtraining/train_lora.py loads Qwen/Qwen3-VL-8B-Instruct with AutoModelForImageTextToText.from_pretrained and attaches a f
training/train_lora.py loads Qwen/Qwen3-VL-8B-Instruct with AutoModelForImageTextToText.from_pretrained and attaches a fresh PEFT LoraConfig (r=32, alpha=64, init_lora_weights=true). This general-purpose base is allowed. training/data_download.sh and the trajectory show the base came from Qwen/Qwen3-VL-8B-Instruct at 0c351dd. training/merge_lora.py adds B@A*alpha/r to those base shards. The model/ inventory has standard Qwen3-VL tensor names and bf16 shapes. No specialist change-captioning checkpoint was used.
✓no reference lookupserver.py only extracts images from the request, builds the prompt and calls the vLLM AsyncLLMEngine with greedy Samplin
server.py only extracts images from the request, builds the prompt and calls the vLLM AsyncLLMEngine with greedy SamplingParams. There are no hash tables, answer caches or held-out data. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE. decode_image contains a defensive urllib fallback for http image URLs, but the harness sends data URLs. The fallback only fetches the request's own image and does not look up answers, so it is not an established violation.
✓actual inference uses reviewed weightsserve.sh runs `exec python3 server.py`, and server.py loads MODEL_DIR=model/ into vLLM. A LoRA adapter is enabled only i
serve.sh runs `exec python3 server.py`, and server.py loads MODEL_DIR=model/ into vLLM. A LoRA adapter is enabled only if a top-level adapter/ exists, and none was shipped; training/adapter/ is never loaded. The trajectory shows the merged model being built on the GPU host (steps 374-401). It was then copied with scp from /dev/shm/model_merged into output/model_new (steps 402-423) and swapped in as output/model (step 425). That gives the 4-shard, 750-tensor Qwen3VLForConditionalGeneration listed in the model/*.safetensors inventories. evidence/merged_sha256.txt records the hashes.
✓harness credentials used only by the harnessThe trajectory has no use of ANTHROPIC credentials, codex, `claude -p` or other harness CLIs, and no OPENROUTER_API_KEY
The trajectory has no use of ANTHROPIC credentials, codex, `claude -p` or other harness CLIs, and no OPENROUTER_API_KEY in the submission. The Spin/OpenRouter scoped-key setup is allowed. The base model download used token=os.environ.get('HF_TOKEN') (steps 148-168), which is allowed for downloads. The shipped serving code uses no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed58m 48s into the runtook 10m 36s
startup 150.6 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 58.7 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to t
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 150.6 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 58.7 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to the max_tokens limit); grading scores either zero
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- Top right: The zoom level changed from `161%` to `90%`.\n- Right sidebar: The \"Interactions\" panel is closed in the second image, while it was open in the first.\n- Main canvas: The \"Add Reward Program\" section is visible in the first im ...[clipped]
- ✓healthy after requests
- 2passed7h 12m into the runtook 8m 47s
startup 201.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 201.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.24 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right sidebar changes from the \"Interactions\" panel to the \"Design\" panel.\n- The zoom level in the top right corner changes from 161% to 90% and the canvas is zoomed out.\n- The rulers along the top and left edges show a wider range ...[clipped]
- ✓healthy after requests
- 3passed7h 35m into the runtook 8m 32s
startup 163.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 163.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.27 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right sidebar changes from the \"Interactions\" panel to the \"Design\" panel.\n- The zoom level in the top right corner changes from 161% to 90% and the canvas is zoomed out.\n- The rulers along the top and left edges show a wider range ...[clipped]
- ✓healthy after requests
- 4passed7h 45m into the runtook 8m 10s
startup 195.6 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 195.6 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.71 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right sidebar changes from the \"Interactions\" panel to the \"Design\" panel.\n- The zoom level in the top right corner changes from 161% to 90% and the canvas is zoomed out.\n- The rulers along the top and left edges show a wider range ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of what was handed in:
Model — Qwen3-VL-8B-Instruct (commit 0c351dd…, apache-2.0) fine-tuned with LoRA r=32/α=64 on 12,800 pairs (WUICC train split + 9,600 self-generated UI pairs), then merged into the bf16 weights. No public validation/test split was ever downloaded or used for any choice.
Serving — vLLM 0.14.1 from 180 vendored wheels installed into /tmp at startup; prefix caching; greedy decoding; 512-token cap; images at a 2,073,600-px budget.
Two throughput findings that changed the design
- The 14 s gate is set-sized by image budget: 3.69M px measured 13.87 s (no margin), 2.07M px 8.86 s.
- The first batch a fresh engine serves is ~2 s slower than every later one — and grading always sends its batch to a fresh server. The selfcheck confirmed this: 12.24 s before, 9.27 s after.
server.pynow runs a full-size warm-up batch inside the FastAPI startup hook;/healthonly goes green afterwards, costing ~4 s of the 500 s startup budget.
Measured (8 pinned cores, exact graded shape 70 req / concurrency 14): 6.39 s cold-start first batch, 3.69 s repeat. On the grading image via frontier selfcheck: startup 195.6 s of 500 s, 9.71 s of 14 s (44% spare), 70/70 responses parsed and non-empty — reported as the digest of the submitted folder.
Quality — 0.864 coverage proxy on 50 synthetic pairs outside the training mix (0.872 at training resolution), and 3 of 5 observations on the one public example: zoom 161%→90%, "canvas zoomed out" and wider rulers described; the cursor/blue outline missed and the sidebar panel named wrongly (Design instead of Prototype). Both misses are documented in evidence/README.md rather than papered over.