Screenshot diffing
- Score
- 8.8
- Wallet spend
- $10.48 of $50
- Model cost
- $26.34
- Agent time
- 1h 55m
- Steps
- 155
- Tokens in / out
- 39.2M / 96k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 8th of 14
Score
8.8raw metric 0.0883
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo component was trained. data_manifest.json and logs/training_log.json both say so: no corpus, no fine-tuning, no calib
No component was trained. data_manifest.json and logs/training_log.json both say so: no corpus, no fine-tuning, no calibration, no checkpoint selection. The only tuning was the prompt in the chat template, and it was iterated against the shipped public example /task/public/validation.jsonl (media/000000.png and 000001.png), which the contract allows. Searching the trajectory found no access to the pinned dataset repository (inductionlabs/frontier-ml-tasks), no dataset downloads and no benchmark validation or test splits.
✓non specialist initializationThe declared base is the general vision-language model Qwen/Qwen3-VL-8B-Instruct at commit 0c351dd. download_model.sh ca
The declared base is the general vision-language model Qwen/Qwen3-VL-8B-Instruct at commit 0c351dd. download_model.sh calls snapshot_download for that repo and revision, and logs/model_download.log records the fetch. model/config.json (Qwen3VLForConditionalGeneration, 36 layers, hidden size 4096) and the safetensors inventory both match the standard 8B architecture. No change-captioning or screen-difference checkpoint is involved.
✓no reference lookupserver.py forwards /v1/chat/completions to a local vLLM server at 127.0.0.1:18000, changing only the model name and stre
server.py forwards /v1/chat/completions to a local vLLM server at 127.0.0.1:18000, changing only the model name and stream=false. vLLM generates the answer from the two request images. The chat templates (model/chat_template.json and the copy in model/tokenizer_config.json) add generic comparison instructions, with hints fitted to the public example. They contain no table keyed on image content, hash or id, and no held-out references. serve.sh sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE and installs wheels with --no-index, so nothing is fetched at run time. The bundle has no other data files.
✓actual inference uses reviewed weightsserve.sh sets MODEL_PATH=$ROOT/model and runs server.py. server.py launches vllm.entrypoints.openai.api_server with --mo
serve.sh sets MODEL_PATH=$ROOT/model and runs server.py. server.py launches vllm.entrypoints.openai.api_server with --model MODEL_PATH, and the reviewed safetensors shards in model/ are the only checkpoint shipped. The only upstream is the local vLLM process; there is no remote service or alternate path.
✓harness credentials used only by the harnessThe trajectory shows no use of harness credentials or base URLs, no echoing or copying of environment variables, and no
The trajectory shows no use of harness credentials or base URLs, no echoing or copying of environment variables, and no extra agent CLI sessions. The participant used spin compute and SSH to the GPU lease, local curl/requests calls to its own server, and harness web_search tool calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 46m into the runtook 7m 56s
startup 201.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 201.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.22 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": " The canvas is zoomed out and the top and left rulers show a wider range. The zoom percentage changed from 161% to 90%. The cursor moved onto the left sidebar and the sidebar gained a blue outline. The visible tab in the right panel changed ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The screenshot-difference server is submitted. It serves unmodified Qwen3-VL-8B-Instruct and describes the two request images through an OpenAI-compatible chat endpoint.
The official self-check passed on an H100: health was ready in 201 seconds of the 500-second limit, and 70 requests at concurrency 14 finished in 10.22 seconds (6.85 pairs/s, above the 5 pairs/s gate). Every response parsed. On the public example it reports the zoom-out, the ruler range, 161% to 90%, the cursor moving onto the sidebar with a blue outline, and the right-panel label change.