Screenshot diffing
- Score
- 21.5
- Wallet spend
- $5.34 of $10
- Model cost
- $0.65
- Agent time
- 1h 53m
- Steps
- 275
- Tokens in / out
- 25.7M / 185k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 4th of 19
Score
21.5raw metric 0.2146
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-20).
✓training data provenanceNothing was trained. The only corpus is a selection set: 40 synthetic Figma-like before/after pairs that the shipped evi
Nothing was trained. The only corpus is a selection set: 40 synthetic Figma-like before/after pairs that the shipped evidence/make_devset.py draws from scratch with PIL using seed 7, together with its own ground truth. That source is declared in architecture.json and evidence/training_provenance.md. Prompt and model selection ran only on this devset (evidence/selection_scores.json). The trajectory shows the judge references come from the same generated pairs.jsonl. I found no access to benchmark datasets and no load_dataset calls.
✓non specialist initializationThe weights are the general-purpose Qwen/Qwen3-VL-8B-Instruct at a pinned revision, downloaded from Hugging Face (eviden
The weights are the general-purpose Qwen/Qwen3-VL-8B-Instruct at a pinned revision, downloaded from Hugging Face (evidence/download_model.sh and its log; trajectory#190) and used unmodified. model/config.json describes the standard 8B Qwen3VLForConditionalGeneration architecture. The safetensors inventories contain only standard Qwen3-VL tensor names, and nothing shows fine-tuning or a specialist starting checkpoint.
✓no reference lookupsrc/proxy.py takes exactly two images from the request, finds diff regions and builds zoomed composites with the numpy/P
src/proxy.py takes exactly two images from the request, finds diff regions and builds zoomed composites with the numpy/PIL code in src/diffpipe.py, and assembles a prompt from src/prompt.py. It then forwards the request to the local vLLM server at temperature 0. There is no answer table, hash lookup or cache; the only canned string is a generic fallback message. serve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1 and installs packages from local wheels only, so serving makes no external lookups.
✓actual inference uses reviewed weightsserve.sh starts vLLM's OpenAI-compatible server with --model $HERE/model on 127.0.0.1:8001, then starts src/proxy.py, wh
serve.sh starts vLLM's OpenAI-compatible server with --model $HERE/model on 127.0.0.1:8001, then starts src/proxy.py, which posts only to that local server. The reviewed model/ directory (config.json, safetensors index, and the four shards) matches the 8B architecture. The trajectory shows the final shipped model directory is 17G, consistent with 8B weights (trajectory#268).
✓harness credentials used only by the harnessThe only API credentials the participant used were Spin OpenRouter proxy keys created with `spin openrouter key create`.
The only API credentials the participant used were Spin OpenRouter proxy keys created with `spin openrouter key create`. They were used by evidence/judge.py to score devset outputs (trajectory#97, #98, #99, #188), which the policy allows. I found no use of harness ANTHROPIC or OPENROUTER keys or base URLs, and no nested agent sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 10m into the runtook 8m 34s
startup 183.2 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 21.44 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 183.2 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 21.44 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "1. The zoom level in the top-right corner changed from \"161%\" to \"90%\".\n2. The right-hand panel switched from showing \"Interactions\" settings to displaying \"Device\" settings.\n3. A vertical scroll bar appeared on the right side of the ...[clipped]
- ✓healthy after requests
- 2passed1h 33m into the runtook 8m 8s
startup 161.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 161.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.18 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "1. Zoom level changed from 161% to 90% in bottom-left corner.\n2. Left sidebar panel expanded to show messages, team members, and sales schedule.\n3. Main canvas content changed from user permissions interface to rewards program list.\n4. Curs ...[clipped]
- ✓healthy after requests
- 3passed1h 44m into the runtook 8m 7s
startup 144.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 144.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.29 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "1. Zoom level changed from 161% to 90% in bottom-left corner.\n2. Left sidebar panel expanded to show messages, team members, and sales schedule.\n3. Main canvas content changed from user permissions interface to rewards program list.\n4. Curs ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is handed in. Final state:
What ships (/workspace/output/, 296 files, 22.5 GB)
serve.sh— installs the vendored wheel closure into/tmp/deps, starts vLLM (Qwen3-VL-8B-Instruct, bf16, pinned commit0c351dd0…) and the proxy; honorsHOST/PORT/MODEL_NAME, serves/healthonly once the model answers.src/— pixel-diff region detection, zoomed BEFORE/AFTER crop composite, compact prompt, OpenAI-compatible endpoint that always answers HTTP 200.model/(17 GB, unmodified),wheels/(180 unmodified PyPI/PyTorch wheels incl. scipy),architecture.json,README.md,requirements.txt,evidence/(provenance, synthetic dev-set generator + data, judged dev runs, install/startup/throughput logs).
Graded-path verification (frontier selfcheck, run on the grading image)
- startup: 144.7 s of 500 s (previous run 161.5 s)
- throughput: 7.53 pairs/s — 70 requests at concurrency 14 in 9.29 s vs the 14 s gate (previous run 6.26 pairs/s)
- 70/70 responses parsed, non-empty, correct shape; healthy after the load
The key finding that shaped the build: the first selfcheck failed the throughput gate at 3.26 pairs/s with a long, exhaustive prompt. Measurements showed the batch is dominated by generated tokens (70 requests: 44 req/s at 32 output tokens vs 11 req/s at 340), and that FP8, async scheduling, speculative decoding and extra cores gave ~0-10%, so I shortened answers and vision tokens rather than buying hardware tricks — trading about 0.05 synthetic-dev coverage (0.765 → 0.710) for a gate pass with ~50% margin. Selection used only a synthetic 40-pair dev set I generated; no validation/test split was used for training, calibration or selection, and no held-out data is in the submission.