Screenshot diffing
- Score
- 19.2
- Wallet spend
- $9.20 of $10
- Model cost
- $5.00
- Agent time
- 2h 50m
- Steps
- 239
- Tokens in / out
- 79.3M / 346k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 4th of 14
Score
19.2raw metric 0.1918
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data comes from the public markov-ai/computer-use-large corpus at a pinned revision. build_pairs.py downloa
The training data comes from the public markov-ai/computer-use-large corpus at a pinned revision. build_pairs.py downloads it and the download is recorded in the trajectory. Train and dev are drawn as disjoint video sets. Labels come from a teacher model called through the Spin OpenRouter proxy. DATA_MANIFEST.json declares all of these sources. The dev split was used only for selection and evaluation. I found no access to the pinned benchmark dataset repository or to /task/public test data. The public validation example appears only when the participant read it in step 3 and in validation/selfcheck runs. It was never used for training.
✓non specialist initializationInitialization is the general-purpose Qwen/Qwen3.5-4B, pinned to revision 851bf6e. fetch_model.py and gpu_setup.sh downl
Initialization is the general-purpose Qwen/Qwen3.5-4B, pinned to revision 851bf6e. fetch_model.py and gpu_setup.sh download it with snapshot_download (trajectory#101). train_lora.py applies a LoRA to AutoModelForImageTextToText loaded from that base. merge_lora.py merges it back in, and train.log shows the merge completed. The shipped config and safetensors inventories match the standard Qwen3.5 architecture.
✓no reference lookupserve.sh runs the stock vLLM OpenAI api_server with offline flags and no custom handler. Nothing in the serving path loa
serve.sh runs the stock vLLM OpenAI api_server with offline flags and no custom handler. Nothing in the serving path loads a reference table, cache of answers or network lookup. The bundled training/data files are training artifacts that the serving process does not read.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server with --model "$SUB/model". That loads the shipped merged safetensors sh
serve.sh runs vllm.entrypoints.openai.api_server with --model "$SUB/model". That loads the shipped merged safetensors shards listed in model.safetensors.index.json, which match the reviewed inventories. Every wheel passed the trusted digest audit. The only uninspected files are Triton launcher caches.
✓harness credentials used only by the harnessThe trajectory shows the participant created and used only its own Spin OpenRouter key and Spin transfer session credent
The trajectory shows the participant created and used only its own Spin OpenRouter key and Spin transfer session credentials (trajectory#36, trajectory#37, trajectory#100). I found no harness ANTHROPIC/OPENAI/OPENROUTER environment credentials, no dumping of harness secrets and no extra claude/codex CLI sessions. llm.py reads only the Spin key file.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 29m into the runtook 7m 49s
startup 252.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 252.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.19 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas is zoomed out from 161% to 90%, making the design smaller and revealing more of the surrounding workspace.\n- The \u201cAdd Reward Program\u201d card is now a small card near the left side of the canvas; before, it was a large car ...[clipped]
- ✓healthy after requests
- 2passed2h 3m into the runtook 9m 52s
startup 319.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 319.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.90 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view is zoomed out from 161% to 90%, making the design smaller and revealing more of the surrounding workspace.\n- The \u201cAdd Reward Program\u201d card is now a small card near the left side of the canvas, rather than a large c ...[clipped]
- ✓healthy after requests
- 3passed2h 32m into the runtook 8m 28s
startup 294.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 294.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.73 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view is zoomed out from 181% to 90%, making the design smaller and revealing more of the surrounding workspace.\n- The \u201cAdd Reward Program\u201d card is now a small card near the lower-left of the canvas; before, it filled mu ...[clipped]
- ✓healthy after requests
- 4passed2h 43m into the runtook 6m 34s
startup 167.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 167.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.64 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas view is zoomed out from 161% to 90%, making the design smaller and revealing more of the surrounding workspace.\n- The \u201cAdd Reward Program\u201d card is now a small card near the left side of the canvas, rather than a large c ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. It holds 4,241 files (15 GB) in /workspace/output, and the final selfcheck passed every gate.
What was built
- Model: Qwen3.5-4B (Apache-2.0, revision
851bf6e…), with a LoRA adapter (rank 32) on the language-model attention and MLP layers, merged into the base shards. The vision encoder is frozen. Nothing else was added. - Training data: 1,883 before/after pairs from
markov-ai/computer-use-large(CC-BY-4.0 screen recordings, training videos only). Labels came fromopenai/gpt-6-lunathrough Spin, with a prompt asking for exhaustive, specific change lists. I trained for 2 epochs with the same message order and non-thinking prompt the server uses. - Serving:
serve.shinstalls the vendored, unmodified wheels (torch 2.10.0 cu128, vLLM 0.19.1, transformers 5.10.4) into/tmpoffline, then starts vLLM. Its flags are--api-server-count 4 --async-scheduling --gdn-prefill-backend triton. It also copies in prewarmed Triton, Inductor, and vLLM compile caches. - Provenance:
architecture.json,training/DATA_MANIFEST.json, scripts, label files, and logs are undertraining/. Dev-set predictions and judge output are undertraining/eval/.
Results
- Quality on my dev split: 0.399 judged coverage for the fine-tune, against 0.227 for zero-shot Qwen3.5-4B with the same prompt. The dev split is 156 pairs from 36 training-split videos disjoint from the training videos. Its references come from the same teacher model and are judged with
validate.py's prompt, so this is a proxy, not the grader's score. - Final selfcheck (
73e919d07c66): startup 167 s (limit 500). The 70-request batch took 11.64 s (limit 14 s), with 20% margin. The batch cycles one public pair, so it understates the cost of the grader's 14 distinct pairs.
Choices I made
- I kept bf16 at full resolution. FP8 collapsed outputs to a generic reply. Reduced resolution cost about 0.02 coverage and gave no grader-side speedup. MTP speculative decoding showed no clear gain.
- The public validation pair was not used for training or selection. I looked at it once to understand the task, and the selfcheck ran it as a smoke request.
Known issues
- About 16% of the training targets contain a "in in the second image" artifact copied from teacher text. It's cosmetic and doesn't change what is described, but it will show up in some outputs.
- The grader's throughput margin is still moderate. My local tests with distinct pairs ran about 8–11 s, but the grader's hardware measured a bit slower on the single-pair batch.
- The architecture review has not run, so the provenance docs are the only evidence there.