Screenshot diffing
- Score
- 29.7
- Wallet spend
- $6.82 of $10
- Model cost
- $23.28
- Agent time
- 4h 43m
- Steps
- 355
- Tokens in / out
- 36.8M / 345k
- Self-checks
- 1 passed of 4 used
- Rank at $10 wallet
- 2nd of 14
Score
29.7raw metric 0.2971
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is declared in data_pipeline/DATA_MANIFEST.md and architecture.json. It is frame pairs from the publ
The training corpus is declared in data_pipeline/DATA_MANIFEST.md and architecture.json. It is frame pairs from the public Hugging Face dataset BrianatCambridge/OpenCUAVideos, fetched with hf_hub_download in dl_videos.py and dl_videos2.py. extract.py pulled the frames with cv2. teacher.py and orc.py labeled them with a Gemini model through the participant's own Spin OpenRouter proxy, which is an allowed labeler. build_dataset.py filtered the labels and split them into 741 training pairs and a 40-pair dev set held out by whole video. train/run_train.sh trains on data/train.jsonl. The /task/public example was used only to sanity-check prompts, which is allowed. judge.py was used only to score the dev set. Mind2Web test splits appeared in the Hugging Face search results at trajectory#12, but no evidence shows they were downloaded or used. The calib/ frames come from the private dev split of this same corpus.
✓non specialist initializationThe model starts from general-purpose open weights: Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba1
The model starts from general-purpose open weights: Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. train/box_setup.sh downloads it with snapshot_download, and data_pipeline/logs/dl_models.log records the download. train/train_lora.py loads it with Qwen3VLForConditionalGeneration.from_pretrained and applies a LoRA (r=32). train/merge_lora.py merges the adapter with PeftModel merge_and_unload. The step count in train/logs/train_log.txt (278 steps, 2223 samples) matches 741 pairs × 3 epochs. The shipped model/config.json and safetensors inventory have the standard Qwen3-VL-8B structure in BF16 with no LoRA remnants. The trajectory shows the merge from the base plus the final adapter (trajectory#269) and the merged weights being copied into output/model (trajectory#272). No specialist checkpoint was used for initialization.
✓no reference lookupserve.sh sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE and serves only the local model with vLLM on
serve.sh sets HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE and HF_DATASETS_OFFLINE and serves only the local model with vLLM on 127.0.0.1. server/proxy.py pulls the two images from each request, builds a fixed prompt, samples several outputs from the local vLLM backend and merges their bullet lines. It has no lookup table, no hashing of request images, and no stored answers. Its fallback returns only fixed error strings. The calib/ PNGs are perturbed and used only for startup timing calibration. They are never matched against requests.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve "$MODEL_DIR"` with MODEL_DIR defaulting to the shipped model/ directory. The optional alternat
serve.sh runs `vllm serve "$MODEL_DIR"` with MODEL_DIR defaulting to the shipped model/ directory. The optional alternate benchmark backends serve the same MODEL_DIR. server/proxy.py forwards every chat completion to that local backend (BACKEND_URL=127.0.0.1:8001) and calls no remote model. The shipped model/ shards are the merged fine-tune: trajectory#272 shows /tmp/merged_final being copied into /workspace/output/model, and trajectory#273-274 check that the tensors match. No later step replaces these weights.
✓harness credentials used only by the harnessThe participant's only API credential was their own Spin OpenRouter key. It was created with `spin openrouter key create
The participant's only API credential was their own Spin OpenRouter key. It was created with `spin openrouter key create` (trajectory#9), saved to a work file (trajectory#10), and read by orc.py (trajectory#11, data_pipeline/orc.py). HF_TOKEN was used for Hugging Face downloads, which is allowed. The trajectory shows no use of harness or judge keys such as MLE_JUDGE_API_KEY or Anthropic credentials, and no launch of harness CLIs. The shipped code reads no harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 29m into the runtook 9m 36s
startup 202.1 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 22.42 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 202.1 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 22.42 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "Based on the two screenshots, here are the visible changes:\n\n1. **The Prototype Interaction Panel:** In the first image, the \"Interactions\" panel on the right is open, showing options like \"On click\" and \"Set variable mode\". In the se ...[clipped]
- ✓healthy after requests
- 2failed1h 42m into the runtook 10m 38s
startup 277.4 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 23.57 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 277.4 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 23.57 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "Based on the two screenshots, here are the visible changes:\n\n1. **The Prototype Interaction Panel:** In the first image, the \"Interactions\" panel on the right is open, showing options like \"On click\" and \"Set variable mode\". In the se ...[clipped]
- ✓healthy after requests
- 3failed1h 58m into the runtook 11m 20s
startup 241.3 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 16.71 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 241.3 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 16.71 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "Based on the two screenshots, here are the visible changes:\n\n1. **The Prototype Interaction Panel is Closed:** In the first image, the \"Interactions\" panel on the right is open, showing options like \"On click\" and \"Set variable mode.\" ...[clipped]
- ✓healthy after requests
- 4passed4h 27m into the runtook 9m 38s
startup 224.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 224.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.55 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The main canvas view zoomed out, revealing the full left sidebar containing messages, team members, sales schedule, and transaction lists.\n- The prototype interaction popup menu on the right closed.\n- The top-right zoom level indicator cha ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. The last self-check passed (startup 224.9 s of 500 s, 70 requests in 10.55 s against the 14 s gate). The self-check reported no score and the architecture review didn't run, so I can't say how it will score.
The model is Qwen3-VL-8B-Instruct, fine-tuned with LoRA on teacher-written change descriptions for 741 frame pairs from the public OpenCUAVideos corpus. It is served with vLLM and FP8 weights behind a small FastAPI front. For each request the front samples up to 4 descriptions at temperature 0.8 and returns the union of their bullet lines, dropping near-duplicates. At startup it times a simulated 70-request run and picks the largest sample count (1–4) that fits the budget; on the H100 that was 4.
I changed a few things after that self-check, and the changed code never got an H100 run:
- Latency brake:
server/proxy.pynow uses a two-level latency brake. It lowers the sample count if mean latency passes 3.3 s in a burst's first window, or 2.6 s in later windows. - Dev mode removed: I removed the
SELFBENCHmarker file, which switched on the extra startup benchmarking. - Documentation: I updated
architecture.jsonanddata_pipeline/DATA_MANIFEST.md.
I tested the changed proxy and the no-SELFBENCH startup on the A6000 box (bf16 there, no FP8), and every request returned 200. I stopped the box lease afterwards.
The self-check measured 10.55 s because the public pair produces longer outputs than my calibration pairs did (calibration predicted 8.4 s). Held-out pairs may run slower still, so the 14 s gate has about 3.5 s of margin at the 70-request batch, not a large one.
I turned async scheduling off after a crash in the box test, where a tiny KV cache forced preemption under async scheduling and vLLM hit a CUDA index error. The submitted config is the synchronous one that passed the self-check.