Screenshot diffing
- Score
- 7.1
- Wallet spend
- $8.82 of $50
- Model cost
- $15.45
- Agent time
- 3h 26m
- Steps
- 389
- Tokens in / out
- 315k / 146k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 11th of 19
Score
7.1raw metric 0.07136
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus has three declared sources: - TESS-Computer/tess-agentnet, train split shards 00000 and 00025 at rev
The training corpus has three declared sources: - TESS-Computer/tess-agentnet, train split shards 00000 and 00025 at revision f5920b25. - xlangai/AgentNet, train jsonl files at revision d76ee50a. - Synthetic UI diffs produced by the shipped renderer. These are declared in data_manifest.json with download commands, and src/prepare_agentnet_data.py shows them fetched with hf_hub_download. logs/agentnet_preparation.log shows the snapshot paths and the 440 extracted pairs, and src/build_merged_dataset.py produces the 786-sample training set. The labels came from a general VLM teacher reached through Spin OpenRouter. The public validation.jsonl was used only for validate.py evaluation runs, not for training.
✓non specialist initializationsrc/train_vlm.py loads the general-purpose Qwen/Qwen3-VL-4B-Instruct at pinned revision ebb281ec, applies a LoRA (r16),
src/train_vlm.py loads the general-purpose Qwen/Qwen3-VL-4B-Instruct at pinned revision ebb281ec, applies a LoRA (r16), then merges it and saves the result. logs/train.log confirms the base load at that revision, a 786-sample run and the merge into output/model_v2. The shipped weight inventories have the standard Qwen3-VL bf16 tensor layout with no leftover LoRA keys. The trajectory shows model_v2 was copied into output/model (trajectory#370-373).
✓no reference lookupsrc/server.py decodes the two submitted images and builds a deterministic pixel-diff collage (src/diff_utils.py). It the
src/server.py decodes the two submitted images and builds a deterministic pixel-diff collage (src/diff_utils.py). It then runs Qwen3-VL prefill and greedy decode, and cleans up the generated text. The code contains no answer tables, hash lookups, reference-file reads or network calls at inference time.
✓actual inference uses reviewed weightsserve.sh sets an offline environment and MODEL_DIR=${ROOT_DIR}/model, then execs src/server.py. That script loads Qwen3V
serve.sh sets an offline environment and MODEL_DIR=${ROOT_DIR}/model, then execs src/server.py. That script loads Qwen3VLForConditionalGeneration.from_pretrained(model_dir), and every request goes through engine.infer, which uses that model. The shipped model/ directory holds the three reviewed safetensors shards. The trajectory shows these files downloaded from the model_v2 output (trajectory#370-373) and served from the submission directory.
✓harness credentials used only by the harnessThe only credential used was a Spin OpenRouter key the participant created themselves (trajectory#63). It was used for t
The only credential used was a Spin OpenRouter key the participant created themselves (trajectory#63). It was used for teacher labeling in src/prepare_agentnet_data.py and as the judge key for validate.py, both allowed. Searches of the trajectory found no use of harness-reserved credentials, no access to the reference task repo and no launches of other harness CLIs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed2h 23m into the runtook 8m 59s
startup 286.8 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 58.6 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to t
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 286.8 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 58.6 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to the max_tokens limit); grading scores either zero
- ✗POST /v1/chat/completions · RuntimeError: timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out
- 2passed3h 21m into the runtook 3m 45s
startup 64.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 64.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.77 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The Figma canvas zooms out in the second image, revealing the sidebar menu items \u201cMessages\u201d, \u201cTeam Members\u201d, \u201cSales Schedule\u201d, and \u201cReward Programs\u201d.\n- The cursor moves from hovering over the \u201cSa ...[clipped]
- ✓healthy after requests