Screenshot diffing
- Score
- 7.9
- Wallet spend
- $3.60 of $10
- Model cost
- $8.17
- Agent time
- 4h 28m
- Steps
- 206
- Tokens in / out
- 39.2M / 212k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 10th of 14
Score
7.9raw metric 0.07949
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancegen_data.py streams only the train splits of HuggingFaceM4/WebSight and osunlp/Multimodal-Mind2Web. It applies programme
gen_data.py streams only the train splits of HuggingFaceM4/WebSight and osunlp/Multimodal-Mind2Web. It applies programmed PIL perturbations and writes templated labels. The generation logs record pinned dataset revisions and train parquet shards (8000 + 2000 pairs). DATA.md and architecture.json declare the same sources. train.py's eval holdout is drawn from the synthetic data (200/100 rows). No search of the trajectory found access to inductionlabs/frontier-ml-tasks, the private held-out folder, or any validation/test split. The public example used for local testing (/tmp/before.png and after.png) is allowed shipped public material.
✓non specialist initializationtrain.py and mergescale.py load the general-purpose Qwen/Qwen3-VL-4B-Instruct, which was downloaded via hf download in t
train.py and mergescale.py load the general-purpose Qwen/Qwen3-VL-4B-Instruct, which was downloaded via hf download in the trajectory, then apply and merge a LoRA adapter. Phase 2 resumes from the phase-1 adapter trained in the same session (train_phase2.log). The shipped model.safetensors has a BF16 Qwen3-VL tensor inventory: 713 tensors from model.language_model.embed_tokens to model.visual.pos_embed. That is consistent with a merged 4B base. No change-captioning specialist checkpoint appears anywhere.
✓no reference lookupproxy.py only injects a generic system prompt with a checklist. It caps max_tokens and forwards to the local vLLM at 127
proxy.py only injects a generic system prompt with a checklist. It caps max_tokens and forwards to the local vLLM at 127.0.0.1:8001. serve.sh sets offline environment variables and serves the local model directory. The only bundled data apart from the model and wheels is two tiny (~2 KB) warmup PNGs used by warmup.py for a local warmup request. No answer tables, no reference data and no network calls at serving time.
✓actual inference uses reviewed weightsserve.sh runs vllm serve "$SUBMIT_DIR/model" on 127.0.0.1:8001 and fronts it with proxy.py, which forwards there. model/
serve.sh runs vllm serve "$SUBMIT_DIR/model" on 127.0.0.1:8001 and fronts it with proxy.py, which forwards there. model/ holds the single reviewed checkpoint, model.safetensors (8,875,719,408 bytes). It was obtained from the S3-synced merged phase-2 output (merge_phase2.log: saved /tmp/diff_ckpt2/merged alpha=32). No alternate or decoy checkpoints are shipped. The self-check log shows the vLLM → warmup → proxy chain running.
✓harness credentials used only by the harnessThe trajectory shows no use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL in participant code. HF_TOKEN was u
The trajectory shows no use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL in participant code. HF_TOKEN was used only for Hugging Face downloads, including on the Spin machine, which is allowed. The Spin CLI was used for compute and S3 transfer. spin openrouter appeared only as a --help invocation. No extra harness CLIs were launched; frontier selfcheck/submit are the task's own tooling.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed4h 17m into the runtook 8m 32s
startup 272.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 272.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 4.19 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- A notification banner reading \"Changes saved\" appeared at the top.\n- The page scrolled down, revealing new content at the bottom.\n- The mouse pointer moved from the center to the left.\n- The zoom level changed from 161% to 90%."}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Qwen3-VL-4B fine-tuned in two LoRA phases on 10k synthetic screen-diff pairs (WebSight + Multimodal-Mind2Web train splits only), served via vLLM 0.12 + a checklist-prompt proxy. Self-check passed on H100: startup 273s/500s, 16.7 pairs/s (gate 5), 70/70 valid responses.