Screenshot diffing
- Score
- 10.5
- Wallet spend
- $27.98 of $200
- Model cost
- $8.10
- Agent time
- 3h 2m
- Steps
- 244
- Tokens in / out
- 26.1M / 332k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 6th of 14
Score
10.5raw metric 0.1051
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe submission generated all of its own training, selection and calibration data. Sources are procedural PIL UIs (traini
The submission generated all of its own training, selection and calibration data. Sources are procedural PIL UIs (training/generate.py), Monaco/Chromium native UIs (training/native_ui.py), edge cases (training/generate_edges.py), a mixture (training/mix.py) and relabeling (training/relabel_native.py). training/provenance.json declares each source with its seeds and commands, and the generation logs and manifests are kept. Development sets are separate seeds of the same generators. Nothing loads a public screen-difference dataset or a held-out validation/test split. The participant read /task/public/validation.jsonl only as a shipped example; it was not used as a training corpus. The gpt-4.1-mini judge only scored the participant's own procedural development pairs.
✓non specialist initializationThe model starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd0. training/download_base.
The model starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd0. training/download_base.py and training/base_download.log record the snapshot_download as 'pretrained initialization'. training/train.py applies LoRA to the language-model linear layers in four stages, then merges the adapters. training/checkpoint_delta_verification.json shows the merged checkpoint has the same tensor keys as the base, with 351 vision tensors bitwise unchanged and sampled language tensors changed, which fits fine-tuning from that general base. The model shard inventories (for example lm_head [151936,4096] BF16) match the Qwen3-VL-8B structure. The 4B model was only tried during development and is not shipped.
✓no reference lookupOn each request server.py decodes the image pair and builds crops and pixel-diff composites (vision.py). geometry.py add
On each request server.py decodes the image pair and builds crops and pixel-diff composites (vision.py). geometry.py adds classical OpenCV hints (text pattern, arrow tracking, blue regions, SIFT/RANSAC scale) as prompt text only. The server then calls vLLM generate at temperature 0. There is no answer table, cache, hash lookup, network call or reference-file read in the serving path, and serve.sh sets an offline environment and installs vendored wheels with --no-index.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py starts AsyncLLMEngine with model=ROOT/'model' using online FP8 quantization, and fall
serve.sh runs server.py. server.py starts AsyncLLMEngine with model=ROOT/'model' using online FP8 quantization, and falls back to the base only if model/config.json is missing. model/config.json is present. The reviewed merged checkpoint is model/model.safetensors.index.json plus model/model-0000{1..5}-of-00005.safetensors. A trajectory startup log shows the server reporting ready with model /workspace/output/model (trajectory#214). Every response comes from local vLLM generation; no remote model service is used.
✓harness credentials used only by the harnessThe harness is Codex. OPENAI_API_KEY appears only as a variable name in an environment listing (trajectory#5) and is nev
The harness is Codex. OPENAI_API_KEY appears only as a variable name in an environment listing (trajectory#5) and is never used by the participant's code. The external judge (training/judge_synthetic.py) uses the participant's own Spin OpenRouter key from artifacts/router.json, which `spin openrouter key create` produced (trajectory#66, trajectory#218). No separate codex or claude worker processes were launched with harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 55m into the runtook 8m 21s
startup 211.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 211.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.40 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The right sidebar's interaction dropdown closes. The canvas zooms out; both zoom indicators change from 161% to 90%, and the artwork becomes smaller."}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with serve.sh, trained weights, offline dependencies, and provenance.
- Trained on 6,000 newly generated screenshot pairs; no public validation/test data used for training or selection.
- Endpoint validator passes.
- Evaluation-infrastructure self-check passes: 211-second startup, 7.45 pairs/second with 14 requests in flight, and 70/70 valid responses.