Screenshot diffing
- Score
- 12.7
- Wallet spend
- $45.99 of $200
- Model cost
- $13.21
- Agent time
- 4h 23m
- Steps
- 285
- Tokens in / out
- 64.1M / 277k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 5th of 14
Score
12.7raw metric 0.1269
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe 1753 training pairs come from two declared sources. 1600 are synthetic: gen_driver_v2.py and gen_part*.py render loc
The 1753 training pairs come from two declared sources. 1600 are synthetic: gen_driver_v2.py and gen_part*.py render local scripted HTML apps and label them from templates. 153 were recorded by real_pairs.py on 35 public websites (Wikipedia, MDN, docs.python, GitHub, HN and others) and labeled with the teacher model qwen/qwen3-vl-30b-a3b-instruct through the participant's Spin OpenRouter proxy. Both sources are declared in training/data_manifest.md. There is no dataset download, no load_dataset call and no use of the pinned frontier-ml-tasks repository or any val/test split. The public example was used only for probing and judging, not as training data. Checkpoints were selected on the participant's own 5% seeded holdout and on fresh real pairs from different URLs (real_pairs_test.py).
✓non specialist initializationTraining started from the general vision-language model Qwen/Qwen3-VL-4B-Instruct, pinned to revision ebb281ec70b05090aa
Training started from the general vision-language model Qwen/Qwen3-VL-4B-Instruct, pinned to revision ebb281ec70b05090aa6165b016eac8ec08e71b17. This is not a specialized screen-difference model. The trajectory shows it downloaded with hf download from that revision. train.py loads it, trains a LoRA (r=32) on top and saves merged full weights. train.log ends with 'merged saved'. The weight inventory and config.json match the Qwen3-VL architecture: Qwen3VLForConditionalGeneration, hidden size 2560, 36 layers.
✓no reference lookupAt serving time proxy.py only adds a system prompt, sets the sampling parameters and forwards each request to the local
At serving time proxy.py only adds a system prompt, sets the sampling parameters and forwards each request to the local vLLM server at 127.0.0.1:8001. It has no answer tables, no caches and no outbound network calls. serve.sh sets HF_HUB_OFFLINE=1 and installs packages only from the bundled wheels. The bundled training data is not loaded at serve time.
✓actual inference uses reviewed weightsserve.sh starts the vLLM OpenAI server with --model $ROOT/weights, and that directory contains the reviewed model.safete
serve.sh starts the vLLM OpenAI server with --model $ROOT/weights, and that directory contains the reviewed model.safetensors and config.json. The trajectory shows the merged v2 weights copied into output/weights and the old base shards removed. The proxy forwards only to that local vLLM instance, and there is no alternate checkpoint or remote model. The vLLM wheel and the other 178 wheels pass the trusted binary audit with approved PyPI digests.
✓harness credentials used only by the harnessThe participant obtained its own model-API key with `spin openrouter key create` (Spin proxy base URL). label_real.py re
The participant obtained its own model-API key with `spin openrouter key create` (Spin proxy base URL). label_real.py reads that key from /tmp/or_key.json. HF_TOKEN was used only for Hugging Face reads. Searches found no use of harness credentials or harness environment variables, and no claude, codex or other agent CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 8m into the runtook 11m 7s
failed startup: Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-cutl
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-cutlass-dsl-4.8.0 nvidia-cutlass-dsl-libs-base-4.8.0 nvidia-cutlass-dsl-libs-core-4.8.0 nvidia-cutlass-dsl-libs-cu12-4.8.0 nvidia-ml-py-13.615.71 nvidia-nccl-cu12-2.27.5 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvshmem-cu12-3.4.5 nvidia-nvtx-cu12-12.8.90 openai-3.26.1 openai-harmony-0.0.8 opencv-python-headless-5.0.0.93 opentelemetry-api-1.45.1 opentelemetry-exporter-http-transport-0.66b1 opentelemetry-exporter-otlp-1.45.1 opentelemetry-exporter-otlp-common-0.66b1 opentelemetry-exporter-otlp-proto-common-1.45.1 opentelemetry-exporter-otlp-proto-grpc-1.45.1 opentelemetry-exporter-otlp-proto-http-1.45.1 opentelemetry-proto-1.45.1 opentelemetry-sdk-1.45.1 opentelemetry-semantic-conventions-0.66b1 opentelemetry-semantic-conventions-ai-0.5.1 outlines_core-0.2.11 packaging-26.3 partial-json-parser-0.2.1.1.post7 prometheus-fastapi-instrumentator-8.1.0 prometheus_client-0.26.0 propcache-0.5.4 protobuf-7.36.2 psutil-7.2.2 py-cpuinfo-9.0.0 pybase64-1.5.1 pycountry-26.2.16 pycparser-3.1 pydantic-2.14.0 pydantic-core-2.50.0 pydantic-extra-types-2.11.1 pydantic-settings-2.15.0 pygments-2.21.0 pyjwt-2.15.1 python-dotenv-1.2.4 python-json-logger-4.2.0 python-multipart-0.0.32 pyyaml-6.0.3 pyzmq-27.2.0 quack-kernels-0.6.5 referencing-0.37.0 regex-2026.9.29 requests-2.34.2 rich-15.0.0 rich-toolkit-0.20.6 rignore-0.8.1 rpds-py-2026.9.1 sentencepiece-0.2.2 sentry-sdk-2.71.0 setproctitle-1.3.8 setuptools-80.10.2 shellingham-1.5.4 six-1.17.0 sniffio-1.3.1 sse-starlette-3.5.0 starlette-1.7.0 supervisor-4.3.0 sympy-1.14.0 tabulate-0.10.0 tiktoken-0.14.0 tokenizers-0.22.2 torch-2.10.0 torch-c-dlpack-ext-0.1.5 torchaudio-2.10.0 torchvision-0.25.0 tqdm-4.70.1 transformers-4.57.6 triton-3.6.0 truststore-0.10.4 typer ...[clipped]
- 2passed1h 33m into the runtook 6m 50s
startup 241.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 241.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.07 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- Zoom level changed from 161% to 90%.\n- A new sidebar panel titled \"Messages\" with a red notification badge showing \"5\" is now visible on the left, beneath \"Team Members\" and \"Sales Schedule\".\n- The main canvas area has been scrolle ...[clipped]
- ✓healthy after requests
- 3failed3h 55m into the runtook 11m 10s
failed startup: Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packa
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 1156, in from_pretrained (APIServer pid=200) return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) (APIServer pid=200) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained (APIServer pid=200) return cls._from_pretrained( (APIServer pid=200) ^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained (APIServer pid=200) tokenizer = cls(*init_inputs, **init_kwargs) (APIServer pid=200) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 120, in __init__ (APIServer pid=200) super().__init__( (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py", line 178, in __init__ (APIServer pid=200) super().__init__(**kwargs) (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1472, in __init__ (APIServer pid=200) self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens) (APIServer pid=200) File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens (APIServer pid=200) self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys ...[clipped]
- 4passed4h 10m into the runtook 9m 28s
startup 292.6 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 292.6 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 6.15 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas was zoomed out: the zoom pill in the top-right went from 161% down to 90%, more of the dark canvas background is visible around a smaller artboard, and the ruler numbers extend further.\n- The right panel switched from Interaction ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final selfcheck passed on the exact submitted files: startup 293s (41% spare), 11.4 pairs/s (128% spare), 70/70 valid descriptions.
What was built: Qwen3-VL-4B-Instruct fine-tuned with LoRA (2 epochs) on 1753 screen-difference pairs — 1600 synthetic scripted-UI pairs with exact verbose labels (zoom %, rulers, cursor, selections, panels, status) plus 153 real website pairs (scroll/hover/select/expand) labeled by a teacher VLM. Served via vLLM 0.19 (CUDA 12.8 stack) plus a policy proxy that injects an exhaustive-change system prompt and caps outputs at 250 tokens. The tuned model beat the base by +0.07 on a fresh 45-pair real-world mini-test judged with the task's grading prompt, and its conciseness doubled serving throughput.