Screenshot diffing
- Score
- 8.7
- Wallet spend
- $109.83 of $200
- Model cost
- $20.13
- Agent time
- 8h 55m
- Steps
- 438
- Tokens in / out
- 43.6M / 465k
- Self-checks
- 1 passed of 3 used
- Rank at $200 wallet
- 7th of 14
Score
8.7raw metric 0.08723
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is a self-generated synthetic set of Playwright-rendered screenshot pairs. architecture.json and dat
The training corpus is a self-generated synthetic set of Playwright-rendered screenshot pairs. architecture.json and data_manifest.json declare it: 60k rendered, 32k trained, seeds 200000-259999. It is produced by build/rend/gen_jobs.py and build/rend/render.py, whose parametric mutators generate the observation targets, then tokenized by build/scripts/prep_data.py. train.log confirms 32000 samples. build/scripts/setup_render.sh also downloads public HTML from the HuggingFaceM4/WebSight train split. This source is declared in shipped code, although data_manifest.json says 'public_data_used: none', which is a minor inconsistency. The download is only partly logged, so WebSight provenance is recorded as unverified. It is a public train split, not held-out or test data. The dev split (seeds 300000-300399) was self-generated and used only to tune serving parameters. No benchmark validation or test data was accessed.
✓non specialist initializationbuild/scripts/run_train.sh downloads the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at the pinned revision cc594898137f
build/scripts/run_train.sh downloads the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at the pinned revision cc594898137f460bfe9f0759e9844b3ce807cfb5. build/scripts/train.py loads it with Qwen2_5_VLForConditionalGeneration.from_pretrained(BASE), freezes the vision tower except the merger, and fully fine-tunes the LLM. train.log shows the 5-shard base loaded and 1334 training steps. The shipped checkpoint (model/config.json and model/model-0000X-of-00004.safetensors) has the standard Qwen2.5-VL tensor layout in BF16, which matches this lineage. Nothing shows initialization from a task-specialist checkpoint.
✓no reference lookupAt serve time, serve.sh starts a local vLLM server on 127.0.0.1:8001, with HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE set a
At serve time, serve.sh starts a local vLLM server on 127.0.0.1:8001, with HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE set and wheels installed with --no-index. proxy.py is a stdlib HTTP proxy that only adds frequency_penalty=0.3 and forwards requests to that local vLLM. It has no answer cache, lookup table, reference data or external network calls.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server with --model "$DIR/model", the shipped checkpoint whose tensors appear
serve.sh runs vllm.entrypoints.openai.api_server with --model "$DIR/model", the shipped checkpoint whose tensors appear in the weight inventory (model/model.safetensors.index.json and the four safetensors shards). proxy.py sends every request to that local server. No other model path or remote model is called. The vLLM wheel and the other wheels have approved trusted binary audits with PyPI provenance.
✓harness credentials used only by the harnessThe session's trajectory shows the participant creating their own Spin OpenRouter key with `spin openrouter key create`
The session's trajectory shows the participant creating their own Spin OpenRouter key with `spin openrouter key create` (trajectory#76) and writing it to a build-only secrets file (trajectory#77). build/scripts/judge_dev.py used that key, through MLE_JUDGE_API_KEY and the Spin base URL, for the dev-split judge. No harness API keys, harness base URLs or extra agent CLI sessions appear in the trajectory or the shipped code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed5h 43m into the runtook 8m 11s
startup 179.6 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 16.97 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 179.6 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 16.97 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- the canvas is panned so that the top left corner of the canvas moves from coordinates (580, 246) to (1793, 10)\n- the active tool changed from \"Hand\" to \"Pen\"\n- the blue selection outline moves from the Sam frame on layer \"Staff access ...[clipped]
- ✓healthy after requests
- 2failed7h 5m into the runtook 8m 42s
startup 182.5 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.19 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 182.5 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 17.19 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- the canvas is panned so that the top left corner of the canvas moves from coordinates (580, 246) to (1793, 10)\n- the active tool changed from \"Hand\" to \"Pen\"\n- the blue selection outline moves from the Sam frame on layer \"Staff access ...[clipped]
- ✓healthy after requests
- 3passed8h 43m into the runtook 8m 27s
startup 187.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 187.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.94 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- the canvas is panned so that the top left corner of the canvas moves from coordinates (60, 125) to (397, 48)\n- the zoom level in figma changes from 161% to 90%\n- a new layer named \"Messages\" appears at the bottom of the layers panel\n- a ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
All verified. Launching selfcheck #3 with the J config: