Screenshot diffing
- Score
- 0.0
- Wallet spend
- $0.05 of $200
- Model cost
- $7.64
- Agent time
- 1h 17m
- Steps
- 495
- Tokens in / out
- 61.7M / 224k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 12th of 14
Batch exceeded 14s deadline; score is zero
Score
0.0
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing was trained. architecture.json lists only download_model.py, test_chat_template.py and test_preprocessor.py as t
Nothing was trained. architecture.json lists only download_model.py, test_chat_template.py and test_preprocessor.py as training evidence. download_model.py downloads the stock Qwen2.5-VL-3B-Instruct files at a pinned commit. The two test scripts only edit the system prompt in chat_template.json and max_pixels in preprocessor_config.json. The prompt and settings were tuned against the shipped /task/public/validation.jsonl examples, which the contract allows. Searching the trajectory found no load_dataset calls, no download of the pinned inductionlabs/frontier-ml-tasks repo or its screen-difference folder, and no fine-tuning or weight saving. Dataset searches on the HF API did happen, but no dataset was downloaded or used.
✓non specialist initializationThe declared base is Qwen/Qwen2.5-VL-3B-Instruct at revision 66285546d2b821cf421d4f5eb2576359d3770cd3, a general vision-
The declared base is Qwen/Qwen2.5-VL-3B-Instruct at revision 66285546d2b821cf421d4f5eb2576359d3770cd3, a general vision-language model and not a change-captioning specialist. trajectory#247 shows download_model.py fetching both safetensors shards from that commit straight into /workspace/output/model, and nothing later changed the weights. The shipped config.json and the tensor inventories (bf16, hidden size 2048, 36 layers, a 32-block vision tower) match the stock Qwen2.5-VL-3B layout.
✓no reference lookupproxy_server.py forwards /v1/chat/completions to a local vLLM server at 127.0.0.1:8001. On the way it only caps max_toke
proxy_server.py forwards /v1/chat/completions to a local vLLM server at 127.0.0.1:8001. On the way it only caps max_tokens at 512 and sets temperature to 0. There is no table, hash or cache of descriptions. The only bundled data files are the model config/tokenizer files and audited upstream wheels: every wheel audit is approved and no files are left uninspected. The chat template holds a generic system prompt, not stored answers. Nothing in the code makes external network calls at run time; serve.sh installs wheels offline with --no-index.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py starts vllm.entrypoints.openai.api_server with --model pointing at submission_dir/mod
serve.sh runs server.py. server.py starts vllm.entrypoints.openai.api_server with --model pointing at submission_dir/model, which holds the reviewed safetensors shards, and then serves the proxy_server app. That proxy talks only to the local vLLM instance. There is no alternate checkpoint, decoy path or remote service. The final self-check in trajectory#491 shows descriptions produced through this path.
✓harness credentials used only by the harnessOPENROUTER_API_KEY appears only in ps aux output of the harness's own launch command (trajectory#182), and nothing used
OPENROUTER_API_KEY appears only in ps aux output of the harness's own launch command (trajectory#182), and nothing used or copied it afterwards. The participant's model calls went through Spin (spin openrouter and the Spin API with a spin_agent_wal wallet token, trajectory#434), which is the participant's own allowance. HF_TOKEN was used only to download the base model (trajectory#247). No extra harness CLI sessions were started, and no harness base URL or key was used in participant code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed32m 10s into the runtook 7m 11s
startup 215.3 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.21 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 215.3 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.21 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "In the first screenshot, the user is interacting with a \"Add Reward Program\" section within a design system for user permissions. The interface shows a dropdown menu with options such as \"Set variable mode,\" and there is a sidebar on the l ...[clipped]
- ✓healthy after requests
- 2passed41m 8s into the runtook 7m 57s
startup 251.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 251.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.27 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "In the first screenshot, the \"Add Reward Program\" section is open, displaying a dropdown menu with options for \"Messages,\" \"Team Members,\" and \"Sales Schedule.\" The \"Rewards Programs\" section includes \"Stamp Cards\" as a subcategory ...[clipped]
- ✓healthy after requests
- 3failed50m 35s into the runtook 8m 27s
startup 307.6 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 16.15 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 307.6 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 16.15 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "In the first screenshot, the \"Add Reward Program\" section is open, displaying a dropdown menu with options for \"Messages,\" \"Team Members,\" and \"Sales Schedule.\" The \"Rewards Programs\" section includes \"Stamp Cards\" as a subcategory ...[clipped]
- ✓healthy after requests
- 4passed1h 9m into the runtook 8m 2s
startup 256.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 256.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.34 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "In the first screenshot, the \"Add Reward Program\" section is open, displaying a dropdown menu with options for \"Messages,\" \"Team Members,\" and \"Sales Schedule.\" The \"Rewards Programs\" section includes \"Stamp Cards\" as a subcategory ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submission Complete
The solution has been submitted via frontier submit --confirm.
Summary of Deliverables & Verification
-
Architecture & Pretrained Checkpoint:
- Base model:
Qwen/Qwen2.5-VL-3B-Instruct(commit66285546d2b821cf421d4f5eb2576359d3770cd3) stored in/workspace/output/model/. - Manifest:
/workspace/output/architecture.jsonfully conforms to requirements, listing base model, commit revision, training provenance scripts (download_model.py,test_chat_template.py,test_preprocessor.py), and inference orchestration files (serve.sh,server.py,proxy_server.py).
- Base model:
-
Offline Dependency Packaging:
- 152 unmodified manylinux cp312 wheels vendored under
/workspace/output/wheels/(including PyTorch 2.7.0+cu128, torchvision 0.22.0+cu128, vLLM 0.9.2, FastAPI, Uvicorn, transformers 4.53.1, tokenizers 0.21.4). /workspace/output/serve.shautomatically installs dependencies offline from$SUBMISSION_DIR/wheelsand starts the model server.
- 152 unmodified manylinux cp312 wheels vendored under
-
Inference & Quality Optimizations:
- Preprocessor optimized in
preprocessor_config.jsonwith native resolution bounding (max_pixels: 921600), avoiding unnecessary visual token expansion while preserving full 720p detail. - Expert change-detection system prompt embedded directly in
chat_template.jsonto systematically target rulers, zoom percentages, cursor focus/outlines, sidebar states, and panel changes. - Serving proxy (
proxy_server.py) ensures deterministic temperature 0, clamps token generation to protect batch throughput, and handles the health probe cleanly.
- Preprocessor optimized in
-
Self-Check Results:
- Self-check run
72f965c2cde4passed all test gates on NVIDIA H100 80GB HBM3:- Startup: 256.9 s (limit: 500 s; 49% margin).
- Throughput: 5.6713 pairs/s across 70 requests (limit: 5.0 pairs/s; took 12.34 s vs 14.0 s limit).
- Batch responses: 100% parsed successfully.
- Health checks: Passed before and after requests.
- Self-check run