Screenshot diffing
- Score
- 38.5
- Wallet spend
- $70.65 of $200
- Model cost
- $8.71
- Agent time
- 5h 21m
- Steps
- 360
- Tokens in / out
- 148.9M / 408k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
38.5raw metric 0.3852
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is declared in provenance/DATA_SOURCES.md and architecture.json as the public HF dataset markov-ai/c
The training corpus is declared in provenance/DATA_SOURCES.md and architecture.json as the public HF dataset markov-ai/computer-use. The retained download manifest lists only data/train-000xx-of-00016.parquet files and recordings/*.mp4 from that dataset; no validation or test files appear. A computer-use-large sample was fetched but, per the documentation, not used. The fetch code is build_manifest_and_fetch.py. split_pairs.py carves the dev split out of these training-source pairs with a hash, and make_pairs_gap2.py keeps only pairs assigned to train. Labels came from Sonnet 5.5 through the Spin OpenRouter endpoint (teacher_label.py) using the participant's own Spin key. I found no access to the pinned frontier-ml-tasks repository or to the held-out set. The public example validation-0001 was used only for smoke and timing tests, never for training.
✓non specialist initializationtrain_launch3.sh calls train_sft.py with --base $HOME/models/Qwen3-VL-8B-Instruct. That path is the general-purpose VLM
train_launch3.sh calls train_sft.py with --base $HOME/models/Qwen3-VL-8B-Instruct. That path is the general-purpose VLM fetched at a pinned revision by fetch_model.py (snapshot_download). train_sft.py loads it with Qwen3VLForConditionalGeneration.from_pretrained, applies LoRA, and merges the adapter in. fp8_convert.py then quantizes only the LM linear layers. From the Qwen3-VL-8B-Instruct-FP8 repo it reads only config.json, to copy quantization_config; it loads no weights from it. The shipped weight inventory has a standard Qwen3-VL layout: an F8_E4M3 weight plus an F32 weight_scale_inv for each LM linear, with BF16 embeddings, lm_head and norms. No specialist change-captioning initialization is involved.
✓no reference lookupproxy.py only forwards requests to the local vLLM server on 127.0.0.1, after forcing the model name, temperature 0 and a
proxy.py only forwards requests to the local vLLM server on 127.0.0.1, after forcing the model name, temperature 0 and a max_tokens cap, and it post-processes output by removing repeated bullets. It contains no answer table, no input hashing and no external network calls. serve.sh sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and VLLM_NO_USAGE_STATS, and installs packages only from the local wheelhouse (--no-index).
✓actual inference uses reviewed weightsserve.sh runs `vllm serve "$HERE/model"` as the only model path, and proxy.py forwards to that backend. The trajectory s
serve.sh runs `vllm serve "$HERE/model"` as the only model path, and proxy.py forwards to that backend. The trajectory shows the packaged model/ directory being replaced with the student3 FP8 checkpoint (trajectory#292, trajectory#293). The file sizes listed there match the reviewed inventory exactly; for example, model-00001-of-00004.safetensors is 3697698104 bytes. The student4 experiment was not shipped (trajectory#349-360 text says student3 was kept).
✓harness credentials used only by the harnessTeacher labeling (teacher_label.py) and dev judging (eval_judge.py) both read the participant's own Spin OpenRouter key
Teacher labeling (teacher_label.py) and dev judging (eval_judge.py) both read the participant's own Spin OpenRouter key from /workspace/work/.or_key.json, which was created with `spin openrouter key create` (trajectory#34). The policy allows this. In the shipped code and the trajectory I found no use of ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, GEMINI_API_KEY, any *_BASE_URL or MLE_JUDGE_API_KEY by the participant, and no HF uploads (push_to_hub). Checkpoints were moved through Spin transfer sessions (s3_transfer.py).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 46m into the runtook 8m 42s
startup 256.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 256.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.81 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- Interactions panel closed, top right.\n- Main canvas: reward program list and table now visible.\n- Left sidebar: \"Messages 5\", \"Team Members\", \"Sales Schedule\" now visible.\n- Left sidebar: \"REWARDS PROGRAMS\" section with \"Stamps C ...[clipped]
- ✓healthy after requests
- 2passed4h 3m into the runtook 8m 57s
startup 271.0 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 271.0 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.34 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- Interactions panel closed, top right.\n- Main canvas: reward table and sidebar now visible.\n- \"Add Reward Program\" card and Sam card moved to the left.\n- Sam card now highlighted with a blue outline.\n- Mouse pointer moved from the right ...[clipped]
- ✓healthy after requests
- 3passed4h 13m into the runtook 8m 14s
startup 256.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 256.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.45 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- Interactions panel closed, top right.\n- Main canvas: reward table and sidebar now visible.\n- \"Add Reward Program\" card and Sam card moved to the left.\n- Sam card now highlighted with a blue outline.\n- Mouse pointer moved from the right ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The solution is submitted. /workspace/output (16.1 GB) is graded as it stands.
What was submitted
- Model: Qwen3-VL-8B-Instruct (Apache-2.0, revision
0c351dd…), fine-tuned with LoRA (rank 32) on 2,357 screenshot pairs for two epochs. The language-model linears are stored as block-wise FP8. The vision tower stays bf16. - Serving:
serve.shinstalls the vendored, unmodified vLLM 0.19.1 and torch 2.10.0 (cu128) wheels into/tmpand runs one engine.proxy.pyin front of it caps output at 200 tokens, drops repeated bullet lines, and retries failures. - Training data: Desktop screenshot pairs from
markov-ai/computer-use: OSWorld per-step pairs, two-step pairs, and 1080p recordings, all regenerated byte-identically from the public files. Targets were super-terse change lists written by Claude Sonnet 5.5 through Spin. Public validation and test splits were not used. The dev split was held out from training. - Evidence:
architecture.json,provenance/(data sources, scripts, labels, logs, adapter, dev outputs).
Verification
frontier selfcheckpassed on the final serving files, twice (checks #2 and #3). Startup took about 257 s of the 500 s allowed. The 70-request batch took 9.45 s against the 14 s limit.- One selfcheck remains unused.
Quality and caveats
- The dev proxy score is 0.607 (0.624 on re-scoring), against 0.767 for the teacher's own descriptions on the same pairs. The proxy judge is
gpt-6-lunawith the grader's prompt, but the references come from Claude Opus, not the grader's data. The judge varies by about ±0.017 between runs, so I can't reliably rank candidates closer than that. - Selection followed the dev results. FP8 on the whole model degenerated, so only the language-model linears are quantized. Two engines on one GPU ran slower than one, and n-gram speculation ran much slower.
- The training labels come from a commercial model. The Spin-provided API was intended for this, and the work is documented. Still, Anthropic's terms may restrict using model outputs for training. A reviewer who disagrees would need the student relabeled.
- The architecture review did not run, so the submission could still score zero on that basis.