Screenshot diffing
- Score
- 19.6
- Wallet spend
- $44.10 of $50
- Model cost
- $24.14
- Agent time
- 9h 47m
- Steps
- 329
- Tokens in / out
- 62.6M / 168k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 5th of 14
Score
19.6raw metric 0.1959
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training corpus has a declared public source: - GUI-World (ONE-Lab/GUI-World): only the Annotation/train/{software
Every training corpus has a declared public source: - GUI-World (ONE-Lab/GUI-World): only the Annotation/train/{software,website}.jsonl files and the train videos they reference were downloaded. The Annotation/benchmark files were listed while exploring the repo but were never used. - showlab/ShowUI-desktop: downloaded with snapshot_download. - Synthetic edits: generated by synth_edits.py, with labels emitted from the edit parameters. - Teacher labels: from qwen/qwen3-vl-235b-a22b-instruct via the participant's Spin openrouter key. The policy allows this. The download commands are kept in download_guiworld.py, download_all.sh and data_manifest.json. The self-built eval set is split by video id from the train data. The task's single public validation pair was used only as a diagnostic score; nothing was trained on it. There is no access to the pinned inductionlabs/frontier-ml-tasks repository and no use of the held-out data.
✓non specialist initializationThe base model is the general VLM Qwen/Qwen3-VL-8B-Instruct, downloaded with snapshot_download; the participant verified
The base model is the general VLM Qwen/Qwen3-VL-8B-Instruct, downloaded with snapshot_download; the participant verified it against revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. train.py loads that base and applies LoRA (r=64). Stage 2 continues from the stage-1 adapter (INIT_FROM=ckpt/run1/lora). merge_lora.py merges the adapter into the same base. The shipped weight inventory has the Qwen3-VL-8B layout (hidden size 4096, 151936 vocab, BF16), and config.json reports Qwen3VLForConditionalGeneration.
✓no reference lookupserver_wrap.py only starts vLLM on the shipped model directory, warms it up with synthetic PIL-drawn images, and proxies
server_wrap.py only starts vLLM on the shipped model directory, warms it up with synthetic PIL-drawn images, and proxies requests to the internal vLLM port, clamping max_tokens. There is no lookup table, no hashing of inputs, no stored reference descriptions and no external network calls at serving time. serve.sh runs fully offline (HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE, pip --no-index).
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and execs server_wrap.py, which runs `vllm serve $SUB/model` with fp8 dyna
serve.sh installs the vendored wheels offline and execs server_wrap.py, which runs `vllm serve $SUB/model` with fp8 dynamic quantization. model/ holds the four reviewed safetensors shards, and their inventory matches the merged Qwen3-VL-8B. The trajectory shows these merged weights being staged, uploaded and copied into /workspace/output/model (steps 308-311), and the selfcheck passed using this stack.
✓harness credentials used only by the harnessThe teacher and judge calls used a scoped key the participant created with `spin openrouter key create` (trajectory step
The teacher and judge calls used a scoped key the participant created with `spin openrouter key create` (trajectory steps 12 and 14), stored in spin_key.txt, with the Spin base URL. The policy allows Spin openrouter calls. HF_TOKEN was copied to the GPU box only for Hugging Face downloads, which the policy allows. Searches found no OPENROUTER_API_KEY or other harness credentials, and the participant launched no extra harness CLIs (claude, codex, etc.).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed9h 16m into the runtook 9m 43s
startup 229.6 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 229.6 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.16 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The 'Interactions' panel on the right is replaced by a 'Design' panel showing 'Device', 'Background', and 'Flows' sections.\n- The 'On click' interaction dropdown is no longer visible.\n- The 'Show prototype settings' button is no longer vis ...[clipped]
- ✓healthy after requests
- 2passed9h 34m into the runtook 9m 17s
startup 199.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 199.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.31 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The 'Interactions' panel on the right is replaced by a 'Design' panel showing 'Device', 'Background', and 'Flows' sections.\n- The 'On click' interaction dropdown is no longer visible.\n- The 'Show prototype settings' button is no longer vis ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is complete and double-verified. Submitting: