Screenshot diffing
- Score
- 26.8
- Wallet spend
- $8.56 of $10
- Model cost
- $31.00
- Agent time
- 3h 25m
- Steps
- 307
- Tokens in / out
- 84.2M / 278k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 3rd of 14
Score
26.8raw metric 0.268
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceAll training pairs come from the public GUI-World dataset (ONE-Lab/GUI-World), train split only. make_pairs.py reads Ann
All training pairs come from the public GUI-World dataset (ONE-Lab/GUI-World), train split only. make_pairs.py reads Annotation/train/*.jsonl and downloads train videos with hf_hub_download. The repo tree in the session showed only Annotation/train/. Teacher labels (seed-2.0-lite) and dev references (gemini-3.8-flash) were generated through the allowed spin OpenRouter broker on GUI-World train-split pairs. The calibration images are also GUI-World train pairs. Nothing shows use of inductionlabs/frontier-ml-tasks beyond /task/public: searches for frontier-ml-tasks and Annotation/test found nothing. /task/public media were used only to test the processor, not for training or selection. The source is declared in architecture.json, README.md and DATA_MANIFEST.md, and its download is recorded in logs/make_pairs.log and the trajectory.
✓non specialist initializationThe base is the official general VLM Qwen/Qwen3.5-35B-A3B-FP8, downloaded at pinned revision 9d1823d2... with hf downloa
The base is the official general VLM Qwen/Qwen3.5-35B-A3B-FP8, downloaded at pinned revision 9d1823d2... with hf download (trajectory#105). It was dequantized to bf16, LoRA-trained with train/train_lora.py on GUI-World teacher labels, and merged back into FP8 shards with train/merge_lora_fp8.py. logs/merge.log records 250 adapted weights across 14 shards. The model/ weight inventory has the same 14 shards. Nothing indicates a specialist screen-difference checkpoint was used as the starting point.
✓no reference lookupproxy.py decodes the request images, builds a prompt and POSTs only to the local vLLM server at 127.0.0.1. On error it r
proxy.py decodes the request images, builds a prompt and POSTs only to the local vLLM server at 127.0.0.1. On error it returns a fixed generic fallback string. It has no answer table, no hash lookup and no remote endpoint. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE and installs wheels offline. The QUALITY table holds dev scores used only to choose the pixel budget. calib/ holds GUI-World train pairs (calib/source_ids.json) used only to warm up and time the server, not as held-out references.
✓actual inference uses reviewed weightsserve.sh runs vllm serve on "$SUB/model" with served-model-name screen-diff. proxy.py forwards every request to that loc
serve.sh runs vllm serve on "$SUB/model" with served-model-name screen-diff. proxy.py forwards every request to that local vLLM instance. The optional MTP speculative decoding uses the same model. The model/ directory was copied from the merged checkpoint (s3 q35-ft, trajectory#198), which logs/merge.log shows as the output of merge_lora_fp8.py. The shipped 14-shard inventory matches that checkpoint. No decoy or alternate checkpoint is in the serving path.
✓harness credentials used only by the harnessThe participant's code calls external models only through a spin OpenRouter key. That key was created with `spin openrou
The participant's code calls external models only through a spin OpenRouter key. That key was created with `spin openrouter key create` (trajectory#27) and is read from SPIN_OR_KEY_FILE or .orkey.json in data_pipeline/orclient.py (trajectory#28). Searches of the session and code found no use of ANTHROPIC_API_KEY, OPENAI_API_KEY or other harness *_BASE_URL credentials, and no extra harness CLI launches. HF_TOKEN was used only for Hugging Face downloads, which is allowed. The anthropic wheel is only a packaged dependency and is not called by the participant's code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 8m into the runtook 15m 46s
startup 264.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 264.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.03 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view zooms out significantly, changing the zoom level from 161% to 90%, revealing a wider section of the design including a left sidebar menu and a transaction table on the right.\n- The floating \"On click\" interaction settings ...[clipped]
- ✓healthy after requests
- 2passed2h 38m into the runtook 12m 44s
startup 240.8 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 240.8 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.15 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view zooms out significantly, changing the zoom level from 161% to 90%, revealing a much wider area of the design file.\n- The floating \"On click\" interaction settings popup that was open in the top-right of the canvas in the fi ...[clipped]
- ✓healthy after requests
- 3passed2h 54m into the runtook 13m 3s
startup 236.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 236.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.15 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right-side panel changes from showing \"Prototype\" interaction settings (with \"On click\" trigger options, variant interactions, scroll behavior controls) to displaying device preview settings for a mobile frame, including Device, View ...[clipped]
- ✓healthy after requests
- 4passed3h 9m into the runtook 13m 30s
startup 280.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 280.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.58 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right-side Properties panel changes from showing Prototype interaction settings (On click, Set variable mode, Scroll behavior options) to showing Device/Frame properties (Device set to None, Background color #000000, Flows list).\n- The ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Ready to submit. The final calibration logic was mock-tested, and every request type it sends was exercised successfully in selfcheck #4.