Screenshot diffing
- Score
- 0.0
- Wallet spend
- $6.92 of $10
- Model cost
- $29.04
- Agent time
- 2h 14m
- Steps
- 59
- Tokens in / out
- 43.7M / 120k
- Self-checks
- 2 passed of 4 used
- Rank at $10 wallet
- 13th of 14
Score
0.0raw metric 0
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo component was trained, calibrated or selected; architecture.json and logs/no_training.txt both say the system is zero
No component was trained, calibrated or selected; architecture.json and logs/no_training.txt both say the system is zero-shot. The only task data used during development was the shipped public before/after example, which policy allows. The participant used it to tune the prompt and check the serving path (trajectory#35, #45, #58). The trajectory shows no download of the pinned dataset repo (inductionlabs/frontier-ml-tasks/screen-difference) or of any other corpus.
✓non specialist initializationarchitecture.json declares the base as the general vision-language model Qwen/Qwen2.5-VL-7B-Instruct at pinned revision
architecture.json declares the base as the general vision-language model Qwen/Qwen2.5-VL-7B-Instruct at pinned revision cc594898137f460bfe9f0759e9844b3ce807cfb5. logs/model_download.txt keeps the snapshot_download command for that revision, and trajectory#59 records the pinned 7B weights being downloaded. The weight inventory matches the Qwen2.5-VL-7B architecture: embed_tokens and lm_head are 152064x3584, MLP is 18944, and GQA k_proj is 512. No fine-tuned or task-specialist checkpoint was shipped or loaded.
✓no reference lookupIn server.py, the /v1/chat/completions handler takes the two screenshots from the request, builds crops and a composite
In server.py, the /v1/chat/completions handler takes the two screenshots from the request, builds crops and a composite from those pixels, and calls AsyncLLM generation with a fixed generic prompt (temperature 0). The code has no answer table, hash lookup or network retrieval. The multimodal processor cache and prefix cache are turned off (mm_processor_cache_gb=0, enable_prefix_caching=False). serve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1. The prompt includes Figma-specific hints taken from the public example but no stored reference descriptions.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=models/Qwen2.5-VL-7B-Instruct and runs `exec python server.py`. server.py loads AsyncLLM with As
serve.sh sets MODEL_DIR=models/Qwen2.5-VL-7B-Instruct and runs `exec python server.py`. server.py loads AsyncLLM with AsyncEngineArgs(model=MODEL_DIR, dtype=bfloat16) and runs every request through ENGINE generation. The shipped safetensors shards listed in the weight inventory are the weights that path loads. The self-check (trajectory#54) confirms the server came up and returned model-generated descriptions.
✓harness credentials used only by the harnessIn the trajectory, the participant used only the harness-provided spin compute CLI and the frontier selfcheck/submit com
In the trajectory, the participant used only the harness-provided spin compute CLI and the frontier selfcheck/submit commands, all through their intended interfaces (trajectory#18, #20, #50, #57). When reading the session files, the participant redacted the secrets (trajectory#11). No step extracts, prints or otherwise misuses a harness credential, and searches for API keys and tokens found nothing improper.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 14m into the runtook 8m 20s
startup 216.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 216.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 5.04 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The screenshots could not be compared (EngineDeadError)."}
- ✓healthy after requests
- 2failed1h 31m into the runtook 9m 18s
startup 198.1 s of 500 s; failed batch responses: VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['validation-0001~0@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"mes
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 198.1 s
- ✗batch responses · VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['validation-0001~0@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'', 'validation-0001~1@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'', 'validation-0001~2@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'', 'validation-0001~3@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'']
- ✗POST /v1/chat/completions · RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 503: b'{"error":{"message":"generation failed","type":"server_error"}}' | RuntimeError: Submission returned HTTP 503: b'{"error":{"message":"generation failed","type":"server_error"}}' | RuntimeError: Submission returned HTTP 503: b'{"error":{"message":"generation failed","type":"server_error"}}'
- 3failed1h 43m into the runtook 8m 43s
startup 211.7 s of 500 s; failed batch responses: VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['validation-0001~0@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"mes
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 211.7 s
- ✗batch responses · VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['validation-0001~0@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'', 'validation-0001~1@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'', 'validation-0001~2@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'', 'validation-0001~3@0: failed 3 attempts: Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\' | Submission returned HTTP 503: b\'{"error":{"message":"generation failed","type":"server_error"}}\'']
- ✗POST /v1/chat/completions · RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 503: b'{"error":{"message":"generation failed","type":"server_error"}}' | RuntimeError: Submission returned HTTP 503: b'{"error":{"message":"generation failed","type":"server_error"}}' | RuntimeError: Submission returned HTTP 503: b'{"error":{"message":"generation failed","type":"server_error"}}'
- 4passed2h 5m into the runtook 8m 48s
startup 211.8 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 211.8 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.14 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The zoom control in Image A is set to 161%, while in Image B it is set to 90%, indicating a zoom out. The right sidebar in Image C is an \"Interactions\" panel, while in Image D it is a \"Prototype\" sidebar. The rightmost ruler number in Imag ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. The server describes visible screenshot changes with Qwen2.5-VL-7B-Instruct, and the last selfcheck passed the runtime gates.
The endpoint takes the before and after screenshots, builds magnified crops of the zoom control, inspector, ruler end, and a labeled side-by-side view, and returns the model’s description. Nothing is looked up from a stored answer.
The selfcheck on an H100 finished startup in 212 seconds (limit 500) and completed 70 requests at concurrency 14 in 11.14 seconds, about 6.3 pairs per second (required 5). The public sample quoted 161% to 90% zoom, the Interactions panel becoming a Prototype sidebar, ruler labels 700 versus 7200, and a blue outline on a canvas frame after the pointer moved onto it.