Screenshot diffing
- Score
- 29.6
- Wallet spend
- $34.64 of $50
- Model cost
- $11.50
- Agent time
- 7h 5m
- Steps
- 294
- Tokens in / out
- 95.1M / 382k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 3rd of 14
Score
29.6raw metric 0.2955
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTraining frames come from public screen recordings in the HF dataset sjhu/GUI_video_new at a pinned revision. This is de
Training frames come from public screen recordings in the HF dataset sjhu/GUI_video_new at a pinned revision. This is declared in architecture.json and data/SOURCES.md, and the corpus has no benchmark split. code/download_sjhu.py downloads the videos with hf_hub_download, and download logs and video lists are kept. code/sample_pairs.py samples frame pairs from those videos. Labels come from an allowed teacher, gpt-5.6-luna via spin OpenRouter (code/label_teacher.py). code/build_sft.py filters the labels and makes the internal dev split, which uses five training-corpus videos excluded from training. The trajectory has no access to inductionlabs/frontier-ml-tasks or held-out data. It also has no use of any public validation or test split: the dataset survey at trajectory#25-27 only inspected candidates, and the one public validation pair was viewed for orientation inside /task/public, which is allowed.
✓non specialist initializationTraining starts from Qwen/Qwen3-VL-8B-Instruct, a general VLM, pinned to revision 0c351dd…. This is declared in architec
Training starts from Qwen/Qwen3-VL-8B-Instruct, a general VLM, pinned to revision 0c351dd…. This is declared in architecture.json, loaded in code/train_sft.py with AutoModelForImageTextToText.from_pretrained, adapted with LoRA, then merged with merge_and_unload. The weight inventory matches the stock Qwen3-VL layout: model.language_model.*, lm_head and visual tower tensors, with no leftover adapter tensors. No change-captioning or screen-difference specialist checkpoint appears in the code or the trajectory.
✓no reference lookupserve.sh only installs the offline wheelhouse, sets the HF and transformers offline flags, and runs the stock vLLM OpenA
serve.sh only installs the offline wheelhouse, sets the HF and transformers offline flags, and runs the stock vLLM OpenAI api_server on $HERE/model. There is no custom serving code, no lookup table, no bundled held-out data and no network call at serve time. The shipped data files are training-provenance artifacts and are never loaded by serve.sh.
✓actual inference uses reviewed weightsserve.sh runs `vllm.entrypoints.openai.api_server --model "$HERE/model"`. That directory holds the five reviewed safeten
serve.sh runs `vllm.entrypoints.openai.api_server --model "$HERE/model"`. That directory holds the five reviewed safetensors shards plus model.safetensors.index.json, all listed in the weight inventory. The trajectory shows these shards were merged at /root/run_v2/merged on the GPU box, checked by SHA-256 against the GPU copy, and moved into /workspace/output/model (trajectory#261-264). Every wheelhouse binary audit is marked approved.
✓harness credentials used only by the harnessThe teacher and judge scripts (code/label_teacher.py, code/judge_dev.py) authenticate with a spin OpenRouter key created
The teacher and judge scripts (code/label_teacher.py, code/judge_dev.py) authenticate with a spin OpenRouter key created through `spin openrouter key create`, which is allowed. HF_TOKEN is used only for HF downloads and API queries (trajectory#19, trajectory#25), which is allowed. The environment listing masks secret values (trajectory#18). The trajectory has no references to ANTHROPIC_API_KEY or OPENROUTER_API_KEY and no claude, codex or other CLI harness launches. The anthropic wheel in the wheelhouse is an approved vLLM dependency and is not called by participant code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 24m into the runtook 8m 36s
startup 268.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 268.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.71 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The prototype view zooms out from 161% to 90%, revealing a wider screen layout.\n- The left user card changes from a large \u201cSam, Owner\u201d card with \u201cAdd Reward Program\u201d above it to a smaller card in a left sidebar, with the ...[clipped]
- ✓healthy after requests
- 2passed6h 14m into the runtook 8m 4s
startup 210.8 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 210.8 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.70 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view changed from a close-up of the \u201cAdd Reward Program\u201d card and user popover to a wider dashboard view, revealing the left sidebar and transaction table.\n- The \u201cAdd Reward Program\u201d card is now positioned nea ...[clipped]
- ✓healthy after requests
- 3passed6h 44m into the runtook 8m 59s
startup 218.3 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 218.3 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.80 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view changed from a close-up of the \u201cAdd Reward Program\u201d card and user popover to a wider dashboard view, revealing the left sidebar and transaction table.\n- The \u201cAdd Reward Program\u201d card is now positioned nea ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output was submitted at 13:01 UTC and will be graded as it stands. Its last self-check passed on that exact content.
What's submitted
- Model: Qwen3-VL-8B-Instruct (revision
0c351dd0), with a LoRA (rank 64) merged into bf16 weights. It was trained for 2 epochs on 4,902 pairs from the publicsjhu/GUI_video_newdesktop screen recordings (CAD and IDE tools). - Labels: descriptions written by
openai/gpt-5.6-lunathrough the spin gateway. Clock, subtitle and playback bullets were filtered out. - Serving:
serve.shinstalls the vendored vLLM 0.19.1 / torch 2.10.0+cu128 wheelhouse offline into/tmp, then serves the weights. It caps the processor at 1,310,720 pixels, so oversized screenshots can't blow the throughput gate. That cap doesn't change 1280×720 or 1280×960 inputs. - Provenance:
architecture.json,data/SOURCES.md, manifests, teacher outputs, download and sampling scripts, and training and serving logs are all in the folder.
Self-checks (the grader's own infrastructure)
- v1 passed: startup 268.7 s, batch of 70 in 11.71 s against the 14 s gate.
- v2 passed: startup 210.8 s, batch in 9.70 s.
- Final (v2 + pixel cap) passed: startup 218.3 s, batch in 10.80 s, 30% margin on throughput.
Proxy quality (internal only) I scored a dev split of five held-out training videos with the coverage judge, using the teacher's bullets as references. This is not the official score, and I couldn't run the official judge.
- Zero-shot base: 0.179
- v1: 0.330 (two judge passes)
- v2: 0.361 (two judge passes)
- Judge noise is about ±0.02.
Things I tried and dropped
- Qwen3.5-9B scored 0.400 on the proxy. Its serving latency under vLLM was too high for the gate (12.3 s uncached in bf16), and both FP8 variants produced garbage output, so I didn't ship it.
Caveats
- The 30% throughput margin is comfortable but not large. The grader's environment varied by about 1 s between self-checks.
- The dataset card lists no license. I used it as a training corpus only and didn't ship any frames.
- I viewed the public validation pair once to understand the task. It was never used for training, labels or model selection.
- I did not use the grader's judge (
gpt-6-luna) anywhere, and my teacher was a different model.