Screenshot diffing
- Score
- 0.0
- Wallet spend
- $34.98 of $50
- Model cost
- $12.70
- Agent time
- 7h 55m
- Steps
- 572
- Tokens in / out
- 60.7M / 403k
- Self-checks
- 1 passed of 4 used
- Rank at $50 wallet
- 13th of 14
4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ["c2bc7f40dc26ed91a5583287@0: failed 3 attempts: Submission returned HTTP 413: b'Maximum request body size 1048576 exceeded.' | Submission returned
Score
0.0
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceAll training data is synthetic and was made by the participant. training/data_gen/sg_gen.py renders HTML screens with Pl
All training data is synthetic and was made by the participant. training/data_gen/sg_gen.py renders HTML screens with Playwright Chromium and builds the captions from the changes it sampled. The logs record 17,776 training pairs and a separate 295-pair validation set. architecture.json declares that no public datasets and no benchmark validation or test splits were used. The trajectory shows the data being regenerated with the participant's own seeds and the run-2 training launched on it. No dataset was pulled from a forbidden source.
✓non specialist initializationTraining starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at a pinned revision. train_lora.py loads that base m
Training starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at a pinned revision. train_lora.py loads that base model and applies LoRA only to the language-model projections, and it raises an error if any vision weights are trainable. merge_lora.py merges the adapter into the base model. The shipped model/config.json is qwen3_vl, and the safetensors inventory has the standard Qwen3-VL tensor layout. The trajectory shows run 2's adapter being merged and that merged model being put into /workspace/output/model. No specialist checkpoint for screen differencing was used.
✓no reference lookupproxy.py is a thin aiohttp reverse proxy. It caps max_tokens (to 120 at the end of the session), adds an optional SYSTEM
proxy.py is a thin aiohttp reverse proxy. It caps max_tokens (to 120 at the end of the session), adds an optional SYSTEM_PROMPT from the environment that is empty by default, and otherwise forwards requests unchanged to the local vLLM server. It has no answer table or reference lookup. serve_warmup.py only sends random synthetic rectangle images. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE.
✓actual inference uses reviewed weightsserve.sh starts vLLM 0.11.0 with --model "$PWD/model" on a localhost-only upstream port. The public port is proxy.py, wh
serve.sh starts vLLM 0.11.0 with --model "$PWD/model" on a localhost-only upstream port. The public port is proxy.py, which forwards only to that upstream. The model directory holds the reviewed qwen3_vl config and the four inventoried safetensors shards listed in the index. Hub access is turned off, so no other model can be loaded.
✓harness credentials used only by the harnessThe only credentials the participant used were allowed tools from its own session. One was an OpenRouter key from `spin
The only credentials the participant used were allowed tools from its own session. One was an OpenRouter key from `spin openrouter key create`, used through ask.py for an optional Gemini teacher. The other was spin-transfer R2 credentials, used to move the participant's own merged model into /workspace/output. Searches found no use of harness-only credentials, the harness BASE_URL, an Anthropic key or claude -p.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed5h 27m into the runtook 12m 24s
startup 233.0 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 26.62 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 233.0 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 26.62 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The zoom level changes from 161% to 90% (the view zooms out)\n- Content appears smaller because of the zoom change.\n- A context menu opens (Cut / Copy / Paste / \u2026)\n- The open context/dropdown menu is closed.\n- A modal dialog (\u201cU ...[clipped]
- ✓healthy after requests
- 2failed7h 12m into the runtook 7m 41s
startup 160.3 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.99 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 160.3 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.99 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The zoom level changes from 161% to 90% (the view zooms out)\n- Content appears smaller because of the zoom change.\n- A context menu opens (Cut / Copy / Paste / \u2026)\n- The open context/dropdown menu is closed.\n- A modal dialog (\u201cS ...[clipped]
- ✓healthy after requests
- 3passed7h 21m into the runtook 6m 29s
startup 125.8 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 125.8 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 13.70 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The zoom level changes from 161% to 90% (the view zooms out)\n- Content appears smaller because of the zoom change.\n- A context menu opens (Cut / Copy / Paste / \u2026)\n- The open context/dropdown menu is closed.\n- A modal dialog (\u201cS ...[clipped]
- ✓healthy after requests
- 4failed7h 34m into the runtook 6m 37s
startup 153.8 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.89 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 153.8 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.89 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The zoom level changes from 161% to 90% (the view zooms out)\n- Content appears smaller because of the zoom change.\n- A context menu opens (Cut / Copy / Paste / \u2026)\n- The open context/dropdown menu is closed.\n- A modal dialog (\u201cS ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Everything validated and in place. Submitting: