Screenshot diffing
- Score
- 11.6
- Wallet spend
- $9.47 of $10
- Model cost
- $18.79
- Agent time
- 8h 22m
- Steps
- 922
- Tokens in / out
- 81.9M / 1.1M
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 8th of 14
Score
11.6raw metric 0.1163
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training and validation data is a synthetic corpus the participant built themselves: 4200 training and 160 validatio
The training and validation data is a synthetic corpus the participant built themselves: 4200 training and 160 validation UI screenshot pairs, rendered with Playwright by the shipped generator (evidence/generator/uigen/generate.py) using fixed seeds set in gen_data.sh. architecture.json and evidence/data_manifest.json declare this source. Generation logs and metadata JSONL files are retained. The generator makes no downloads and does not read /task. The session shows /task/public was used only for development checks (trajectory#67), and nothing shows the benchmark dataset being used as training data.
✓non specialist initializationThe model starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct, downloaded from HF at a pinned revision in setup.sh
The model starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct, downloaded from HF at a pinned revision in setup.sh. train_sft.py loads it with Qwen3VLForConditionalGeneration.from_pretrained, applies PEFT LoRA to the language-model projection layers, then runs merge_and_unload and saves to merged/. model/config.json and the safetensors tensor inventories match the standard Qwen3-VL-8B structure: 36 text layers, a 27-layer vision tower, no leftover adapter tensors and no extra checkpoints. trajectory#873 shows training finished and the merged model was saved.
✓no reference lookupserver/server.py decodes the two input images, crops them to the differing regions with uigen/preprocess.py (pure image
server/server.py decodes the two input images, crops them to the differing regions with uigen/preprocess.py (pure image diffing), builds a fixed prompt with uigen/prompting.py, and sends the request to the local vLLM backend at temperature 0. The serving code has no answer table, hash lookup, cache of references, or outbound network call.
✓actual inference uses reviewed weightsserve.sh starts the vLLM OpenAI server with --model $SCRIPT_DIR/model on 127.0.0.1:8001. The server/server.py proxy send
serve.sh starts the vLLM OpenAI server with --model $SCRIPT_DIR/model on 127.0.0.1:8001. The server/server.py proxy sends every chat completion to that local backend. The shipped model/ directory holds the reviewed merged Qwen3-VL bf16 safetensors. train.sh copies run/merged to the model directory, and the session shows the trained model tarball being retrieved and assembled into the output (trajectory#872, trajectory#900).
✓harness credentials used only by the harnessThe session shows no use of harness-reserved credentials. The only model-API calls went through the participant's allowe
The session shows no use of harness-reserved credentials. The only model-API calls went through the participant's allowed Spin OpenRouter proxy key (trajectory#68, trajectory#69, trajectory#71). Compute and storage used Spin GPU and Spin R2.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed8h 11m into the runtook 9m 12s
startup 226.2 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 226.2 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.10 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas is zoomed out: the zoom level in the top-right corner drops from 161% to 90%, and the frames appear smaller with more empty space around them.\n- The right sidebar switches from the \u201cPrototype\u201d tab to the \u201cDesign\u2 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Complete pass on the real grading H100:
| Gate | Result | Requirement | Margin |
|---|---|---|---|
| Startup | 226.2s | 500s | 55% spare |
| Throughput | 6.93 pairs/s (10.10s / 70 reqs) | 5 pairs/s (14s) | 39% spare |
| Responses | 70/70 parsed, non-empty | 70 | ✓ |
| Single request | 1.27s + high-quality description | — | ✓ |
Submitting: