Screenshot diffing
- Score
- 10.0
- Wallet spend
- $17.46 of $50
- Model cost
- $21.45
- Agent time
- 3h 51m
- Steps
- 474
- Tokens in / out
- 45.6M / 280k
- Self-checks
- 1 passed of 3 used
- Rank at $50 wallet
- 9th of 19
Score
10.0raw metric 0.1002
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data is synthetic only. data_manifest.json declares the train_v3_0..11 sets (500 pairs each) and browser_tr
The training data is synthetic only. data_manifest.json declares the train_v3_0..11 sets (500 pairs each) and browser_train (1,200 pairs). These were generated by the shipped Pillow and Playwright/Chromium generators (training/reproduce_data.sh, training/generate_synthetic.py), and their labels come from the scripted state transitions. training/train_sft.py reads only the --data jsonl globs. The train run in the trajectory used data/train_v3_* and data/browser_train. Checkpoint selection used separately seeded synthetic dev sets (dev_v3, browser_dev_v2, stress_v3). The session only looked up public datasets on the web and never downloaded any or accessed reference/test data.
✓non specialist initializationThe base model is the general VLM Qwen/Qwen3.5-9B at pinned revision c202236235762e1c871ad0ccb60c8ee5ba337b9a. The downl
The base model is the general VLM Qwen/Qwen3.5-9B at pinned revision c202236235762e1c871ad0ccb60c8ee5ba337b9a. The download is logged in logs/base_model_download.log via snapshot_download. training/train_sft.py loads the model with AutoModelForImageTextToText.from_pretrained and applies LoRA to the language modules only. training/merge_lora.py merges the adapter into that same base (trajectory#436 used --base /workspace/output/model) and restores the base mtp tensors. The tensor inventory in model/model-00001-of-00004.safetensors matches the standard Qwen3.5 structure.
✓no reference lookupserver.py only proxies /v1/chat/completions to a local vLLM engine on 127.0.0.1:8001. It adds a fixed system prompt, for
server.py only proxies /v1/chat/completions to a local vLLM engine on 127.0.0.1:8001. It adds a fixed system prompt, forces temperature 0 and caps max_tokens. It has no lookup tables, answer caches or remote calls, and urllib is used only for the localhost health check. serve.sh sets HF_HUB_OFFLINE and installs only from the shipped wheels/.
✓actual inference uses reviewed weightsserve.sh runs server.py with --model $ROOT/model. server.py starts vllm.entrypoints.openai.api_server with that model pa
serve.sh runs server.py with --model $ROOT/model. server.py starts vllm.entrypoints.openai.api_server with that model path. The model/ safetensors shards are in the reviewed weight inventory. logs/merge_checkpoint200_comparison.json shows all 16 files of the local and remote merges match by hash. No other model endpoint is used at serving time.
✓harness credentials used only by the harnessThe judge credential was created with the permitted `spin openrouter key create` (trajectory#78, trajectory#214). It was
The judge credential was created with the permitted `spin openrouter key create` (trajectory#78, trajectory#214). It was used through SPIN_JUDGE_KEY and SPIN_JUDGE_BASE for the OpenRouter-proxy judge (training/judge_own.py, trajectory#97), which is allowed. The session shows no use of ANTHROPIC_API_KEY or another harness credential from participant code; the only such match is a process listing. No extra harness CLI sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed51m 58s into the runtook 11m 39s
failed startup: Server did not become healthy in 500s (last /health probe: HTTP 503 b'{"status":"starting"}'); end of its log: ps-v1/vllm/entrypoints/launchers/api_server/entry.py", line 58, in build_async_engine_client async with build_async_engine_client_from_engine_args( File "/usr/
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: HTTP 503 b'{"status":"starting"}'); end of its log: ps-v1/vllm/entrypoints/launchers/api_server/entry.py", line 58, in build_async_engine_client async with build_async_engine_client_from_engine_args( File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/screen-difference-deps-v1/vllm/entrypoints/launchers/api_server/entry.py", line 92, in build_async_engine_client_from_engine_args async_llm = AsyncLLM.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/screen-difference-deps-v1/vllm/v1/engine/async_llm.py", line 248, in from_vllm_config return cls( ^^^^ File "/tmp/screen-difference-deps-v1/vllm/v1/engine/async_llm.py", line 180, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/screen-difference-deps-v1/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/screen-difference-deps-v1/vllm/v1/engine/core_client.py", line 167, in make_async_mp_client return AsyncMPClient( ^^^^^^^^^^^^^^ File "/tmp/screen-difference-deps-v1/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/screen-difference-deps-v1/vllm/v1/engine/core_client.py", line 1100, in __init__ super().__init__( File "/tmp/screen-difference-deps-v1/vllm/v1/engine/core_client.py", line 674, in __init__ with launch_core_engines( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/tmp/screen-difference-deps-v1/vllm/v1/engine/utils.py", line 1268, in launch_core_engines wait_for_engine_startup( File "/tmp/screen-difference-deps-v1/vllm/v1/engine/utils.py", line 1348, in wait_for_engine_s ...[clipped]
- 2failed1h 19m into the runtook 11m 32s
startup 363.5 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 15.01 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 363.5 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 15.01 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "Here are the visible changes between the two screenshots:\n\n- **Main Canvas Content:**\n - The view has scrolled up and to the left.\n - The \"Add Reward Program\" section is now part of a larger list.\n - New UI elements are visible ...[clipped]
- ✓healthy after requests
- 3passed3h 41m into the runtook 7m 50s
startup 261.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 261.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 4.33 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The zoom read in the top-right corner changes from 161% to 90%, and the main content scales up.\n- The properties panel on the right is hidden, and the main content expands left."}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-24 04:38:16 UTC.
Final verification:
- Offline H100 self-check: passed.
- Startup: 261.5 seconds of the 500-second limit.
- Throughput: 70 requests at concurrency 14 in 4.33 seconds—16.15 pairs/s.
- Responses: all 70 parsed and non-empty.
- Required validator: 0 failed requests.
- Final artifact audit: passed for all 319 files, including model shards, evidence, source, and vendored wheels.