Screenshot diffing
- Score
- 1.6
- Wallet spend
- $5.41 of $50
- Model cost
- $16.91
- Agent time
- 8h 30m
- Steps
- 269
- Tokens in / out
- 45.2M / 175k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 12th of 14
Score
1.6raw metric 0.01643
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus has a declared, retained source. DATA_MANIFEST.md and fetch_base.py both point to the HuggingFaceM4/
The training corpus has a declared, retained source. DATA_MANIFEST.md and fetch_base.py both point to the HuggingFaceM4/WebSight train split: the script keeps only data/train-*.parquet shards, and fetch.log lists 71 train shards and 3500 saved images. gen_data.py uses PIL to apply synthetic edits to those screenshots, producing 6000 pairs with their observations. train.py and train.log show the model was trained on that train.jsonl. Model selection used only training loss, the public /task pair and seed-99 synthetic pairs (trajectory#40, trajectory#147). I found no access to held-out or benchmark data.
✓non specialist initializationThe model starts from the general-purpose Qwen/Qwen2.5-VL-3B-Instruct. architecture.json pins revision 66285546, and the
The model starts from the general-purpose Qwen/Qwen2.5-VL-3B-Instruct. architecture.json pins revision 66285546, and the trajectory shows that revision being looked up and downloaded with snapshot_download (dl_model.py, dl_model.log, trajectory#36). train.py adds LoRA adapters to this base and merges them into full weights. model/config.json is Qwen2_5_VLForConditionalGeneration with _name_or_path /root/base_model. The safetensors inventory contains only standard Qwen2.5-VL BF16 tensors. The shipped model is the one trained from this base (trajectory#236, trajectory#240).
✓no reference lookupserve.sh sets an offline environment and installs vendored wheels with --no-index. It then runs only the vLLM OpenAI api
serve.sh sets an offline environment and installs vendored wheels with --no-index. It then runs only the vLLM OpenAI api_server on $PWD/model. It has no retrieval, cache, answer table or remote call. The model directory holds only the config, tokenizer and processor files and the weight shards.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server with --model "$PWD/model", which serves the inventoried safetensors sha
serve.sh runs vllm.entrypoints.openai.api_server with --model "$PWD/model", which serves the inventoried safetensors shards. The vllm-0.10.2 wheel and all other vendored wheels have approved trusted binary audits. The trajectory shows the final merged v1 model, which matches train.log's 420 steps, being placed in output/model (trajectory#235, trajectory#240). The failed v2 save was not shipped.
✓harness credentials used only by the harnessThe trajectory shows the participant using only a key they created themselves with spin openrouter key create, for the j
The trajectory shows the participant using only a key they created themselves with spin openrouter key create, for the judge (MLE_JUDGE_API_KEY), and HF downloads, both of which are allowed (trajectory#150, trajectory#151). I found no use of the harness's API key or base URL, and no access to the task repository.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed7h 38m into the runtook 7m 39s
startup 202.1 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.32 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 202.1 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.32 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The cursor moves from the center area to the middle-left area of the screen.\n- A left sidebar panel opens showing 'Messages' with 5 items and 'Team Members', 'Sales Schedule', 'Rewards Programs'.\n- The round status indicator in the center ...[clipped]
- ✓healthy after requests
- 2passed8h 20m into the runtook 6m 57s
startup 197.8 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 197.8 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.92 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The round status indicator in the center area changes from blue to green.\n- A left sidebar panel opens showing 'Messages' with options including 'Profile', 'Settings', 'Help', 'Sign out'.\n- The view scrolls down: the page content moves up ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of the delivered solution in /workspace/output/:
Model — LoRA fine-tune (r=64, 420 steps) of Qwen/Qwen2.5-VL-3B-Instruct @ 66285546d2b821cf421d4f5eb2576359d3770cd3, merged to bf16 and served by vLLM 0.10.2 (torch 2.8.0+cu128, vendored wheels for offline install) through serve.sh, with prefix caching and an 802,816 max-pixel cap.
Training data (provenance in DATA_MANIFEST.md) — 6,000 synthetic screenshot pairs generated by gen_data.py: programmatic UI edits (cursor moves, selections, panels, dialogs, text/value changes, layout shifts, scroll, zoom, status/toggles/progress, tabs, typing, no-change) applied to screenshots from the HuggingFaceM4/WebSight training split only (fetch_base.py). No validation/test split of any benchmark was used for training, calibration, or selection; the public example pair was only a qualitative smoke check.
Measured results
frontier selfcheck(H100, offline, eval-identical): passed — startup 197.8s of 500s, throughput 70 requests at concurrency 14 in 8.92s (7.84 pairs/s vs 5 required), all 70 responses valid.- Judge-scored diagnostics (gpt-6-luna, same prompt as the grader): 0.70–0.78 reference-observation coverage on held-out synthetic pairs (seed-disjoint), 0.2 on the single public Figma pair — the model reliably reports zoom/cursor/panel changes but often misses exact readouts (e.g. "161% → 90%") and ruler ranges, which is the main quality headroom left.
A second fine-tuning round with composite design-tool edits was attempted but degenerated into repetition on real screenshots, so the verified v1 model was shipped instead.