Screenshot diffing
- Score
- 0.0
- Wallet spend
- $27.92 of $200
- Model cost
- $1.75
- Agent time
- 7h 29m
- Steps
- 637
- Tokens in / out
- 61.5M / 465k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 15th of 19
Batch exceeded 14s deadline; score is zero
Score
0.0
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data is a synthetic corpus built locally. Headless Chromium renders mock HTML pages (code/mockpages.py), an
The training data is a synthetic corpus built locally. Headless Chromium renders mock HTML pages (code/mockpages.py), and the labels come from DOM mutations. Some public web pages listed in data_manifest.json are only rendered. code/prep_data2.sh splits the generated parts into train (parts 3,4,5,6,7,8,10,11,13,14,15) and a held-out synthetic eval (parts 1,2,9,12). code/train_v2.sh trains only on /root/data2/train.jsonl. Choosing v2 over v1 used that held-out synthetic eval, not any public validation/test split. The trajectory shows no access to the inductionlabs/frontier-ml-tasks repo or any public dataset. /task/public/validation.jsonl was read only to check the format. The listed public URLs have no retained fetch log, so that part is declared but unverified, which the policy accepts.
✓non specialist initializationThe base model is Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b, a general VLM. It was
The base model is Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b, a general VLM. It was downloaded with code/dl_mirror.py. code/train_sft.py applies a LoRA to it, and code/merge_lora.py merges the adapter. In the trajectory, the final merge (pipeline_v3) uses --base /root/models/Qwen3-VL-8B-Instruct with the lora8b_v2 adapter. model/config.json and the tensor inventory show the standard Qwen3-VL-8B architecture. No specialist change-captioning checkpoint was used.
✓no reference lookupserve.sh runs vLLM generation on model/ with HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and PIP_NO_INDEX set, and installs fro
serve.sh runs vLLM generation on model/ with HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and PIP_NO_INDEX set, and installs from local wheels only. model/chat_template.jinja is the standard Qwen3-VL template plus a default system prompt, with no lookup logic. data/ holds only synthetic sample pairs and data/sample.json. No answer tables, held-out data or network calls were found in the serving path.
✓actual inference uses reviewed weightsserve.sh calls `vllm serve $HERE/model` with fp8 quantization applied at load time and the shipped chat template. The sh
serve.sh calls `vllm serve $HERE/model` with fp8 quantization applied at load time and the shipped chat template. The shipped shards are standard Qwen3-VL BF16 tensors, and their sizes match logs/merge_v2.log (for example, model-00001 is 4998056552 bytes). The trajectory shows that exact merged model being downloaded and moved into output/model (package_v2.sh). There is no alternate model path or remote endpoint.
✓harness credentials used only by the harnessSearching the trajectory for ANTHROPIC, BASE_URL and harness CLI launches found nothing relevant. The only key reference
Searching the trajectory for ANTHROPIC, BASE_URL and harness CLI launches found nothing relevant. The only key references are SPIN_API_KEY and spin s5cmd session files used for R2 transfers, plus a mention of MLE_JUDGE_API_KEY in docs. Spin use is allowed. No extra claude or codex sessions were launched, and serve.sh uses no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed4h 6m into the runtook 7m 52s
startup 211.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 211.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 7.98 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The second screenshot differs from the first in these ways:\n- the pointer is now at the centre of the screen instead of the centre of the screen\n- the right sidebar was narrowed, which pushed the neighbouring content sideways\n- the right si ...[clipped]
- ✓healthy after requests
- 2passed6h 46m into the runtook 8m 35s
startup 212.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 212.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.74 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The following differences are visible between the before and after screenshots:\n- the pointer is now at the centre of the screen instead of the centre of the screen\n- the right sidebar (middle-right) narrowed by about 74 pixels, shifting the ...[clipped]
- ✓healthy after requests
- 3passed7h 18m into the runtook 8m 25s
startup 195.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 195.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.09 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The following differences are visible between the before and after screenshots:\n- the pointer is now at the centre of the screen instead of the centre of the screen\n- the right sidebar (middle-right) narrowed by about 74 pixels, shifting the ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Everything is staged. Waiting on training (step 105/550, ETA ~02:08):