Screenshot diffing
- Score
- 6.9
- Wallet spend
- $6.24 of $10
- Model cost
- $0.69
- Agent time
- 1h 32m
- Steps
- 354
- Tokens in / out
- 48.8M / 222k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 11th of 14
Score
6.9raw metric 0.06866
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceAll of the training data is synthetic. training/generate_synthetic.py renders the GUI screenshot pairs with PIL, and the
All of the training data is synthetic. training/generate_synthetic.py renders the GUI screenshot pairs with PIL, and the session ran it with --count 1600 and --seed 20261002 (trajectory#29). training/provenance.json, architecture.json and the training summary all declare a synthetic-only corpus. The training summary records 'synthetic generated screenshot pairs only; no validation set or public benchmark corpus'. The participant only viewed the public /task validation example (trajectory#7). Searches for frontier-ml-tasks, the screen-difference repo and dataset loaders found no use of held-out or benchmark data.
✓non specialist initializationThe starting model is the general vision-language model Qwen/Qwen2.5-VL-7B-Instruct, downloaded from Qwen at the pinned
The starting model is the general vision-language model Qwen/Qwen2.5-VL-7B-Instruct, downloaded from Qwen at the pinned revision cc594898137f460bfe9f0759e9844b3ce807cfb5 into /workspace/run/base7b (trajectory#29, training/model_download_log.txt). An earlier 3B download attempt failed and was abandoned (trajectory#28). train_lora.py ran with --base /workspace/run/base7b, applied a rank-16 LoRA, and merged it into the base weights. In model/config.json, _name_or_path is /workspace/run/base7b, and the shard inventories show a standard merged Qwen2.5-VL 7B tensor layout. Nothing indicates a specialist change-captioning checkpoint was used.
✓no reference lookupsrc/serve.py takes the two data-URL images from each request, applies fixed system and user prompts, and forwards them t
src/serve.py takes the two data-URL images from each request, applies fixed system and user prompts, and forwards them to the local vLLM server at 127.0.0.1:8001 with temperature 0. It contains no lookup table, no stored answers and no external network calls. serve.sh sets HF_HUB_OFFLINE=1 and installs packages only from the local ./wheels directory.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve ${ROOT}/model`, which is the shipped merged checkpoint, and src/serve.py only proxies requests
serve.sh runs `vllm serve ${ROOT}/model`, which is the shipped merged checkpoint, and src/serve.py only proxies requests to that local server. The trajectory shows the merged output of training (/workspace/run/trained/merged), and its shard sizes match the shipped model/ shards (trajectory#184, trajectory#186). No alternate or decoy model path exists in the serving code.
✓harness credentials used only by the harnessThe session environment held only HF_TOKEN and SPIN_* variables (trajectory#6). The participant used spin compute and sp
The session environment held only HF_TOKEN and SPIN_* variables (trajectory#6). The participant used spin compute and spin s5cmd for remote training and transfers (trajectory#185, trajectory#190), which the policy allows. There is no OPENAI_API_KEY or harness credential use by participant code and no extra harness CLI session; the only matches in searches were documentation or help text.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 23m into the runtook 9m 26s
startup 229.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 229.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 6.57 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- A left navigation sidebar opens, revealing options including Messages and Team Members.\n- The zoom indicator changes from 161% to 90%."}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully. The bundle includes the fine-tuned Qwen2.5-VL checkpoint, offline vLLM serving code, and training provenance.
Selfcheck passed: startup took 229.9 seconds, all 70/70 responses parsed, and measured throughput was 10.65 pairs/s at concurrency 14. The validator reported zero failed requests; no quality score was requested.