Screenshot diffing
- Score
- 33.5
- Wallet spend
- $43.14 of $50
- Model cost
- $20.08
- Agent time
- 4h 43m
- Steps
- 269
- Tokens in / out
- 25.9M / 218k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 2nd of 14
Score
33.5raw metric 0.3354
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json and DATA_MANIFEST.md declare the training corpora as public Hugging Face datasets. They are markov-ai/
architecture.json and DATA_MANIFEST.md declare the training corpora as public Hugging Face datasets. They are markov-ai/computer-use-large (all categories) and ONE-Lab/GUI-World, restricted to videos in its own Annotation/train split. training/data/videos.jsonl lists the source videos, training/extract.py downloads them with HF_TOKEN and cuts frame pairs, and select_pairs.py selects pairs from them. The trajectory agrees. At step 20 the GUI-World benchmark annotations were opened only to count rows. At step 25 videos.jsonl was built from Annotation/train only. The dev sets used for checkpoint selection were cut from separate training-split videos. I found no download of the task's reference or test data and no use of a public val/test split. Labels came from claude-sonnet-5.5 through the allowed Spin gateway.
✓non specialist initializationtraining/train_sft.py loads Qwen3VLForConditionalGeneration from Qwen/Qwen3-VL-8B-Instruct, a general-purpose instructio
training/train_sft.py loads Qwen3VLForConditionalGeneration from Qwen/Qwen3-VL-8B-Instruct, a general-purpose instruction-tuned VLM pinned to a revision in architecture.json. It applies LoRA to the language-model linear layers and merges the result. training/convert_fp8.py then quantizes those linear layers to blockwise FP8. The inventory of model/model.safetensors matches: Qwen3-VL language-model layers in F8_E4M3 with weight_scale_inv tensors, BF16 embeddings and lm_head, hidden size 4096. In trajectory step 164-192 the shipped model directory is copied from fp8_ft1, the run1 fine-tune, which matches the declared final model. I saw no task-specialist checkpoint.
✓no reference lookupsrc/proxy_server.py only forwards the request's chat messages to the local vLLM /v1/chat/completions endpoint, with temp
src/proxy_server.py only forwards the request's chat messages to the local vLLM /v1/chat/completions endpoint, with temperature 0 and a capped max_tokens, then trims truncated output. It has no tables keyed on image content, stores no reference answers and makes no remote calls. serve.sh sets HF_HUB_OFFLINE=1 and uses only local wheels and the local model.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve "$HERE/model" --served-model-name submission` on port 8001 and puts the proxy on $PORT in fron
serve.sh runs `vllm serve "$HERE/model" --served-model-name submission` on port 8001 and puts the proxy on $PORT in front of it. The proxy sends every completion to that local vLLM server. The weights served are model/model.safetensors, the reviewed FP8 Qwen3-VL checkpoint, together with model/config.json.
✓harness credentials used only by the harnessAt trajectory step 12 the participant made its own Spin OpenRouter gateway key with `spin openrouter key create`. Step 1
At trajectory step 12 the participant made its own Spin OpenRouter gateway key with `spin openrouter key create`. Step 13 saved it to /workspace/dev/or_key.json, and training/orclient.py uses it for teacher labeling through the Spin gateway. That is the permitted route. I found no use of the harness's Anthropic or other harness API keys in participant code. Step 259 confirms the key values do not appear anywhere in the shipped output.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 55m into the runtook 6m 40s
startup 178.7 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 178.7 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.76 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zoom changed from 161% to 90%, so the whole view is smaller and shifted. The \"Add Reward Program\" card, the Sam/Owner avatar card and the Alex/Staff popover all sit lower and further left.\n- The right-hand Prototype panel is no ...[clipped]
- ✓healthy after requests
- 2passed4h 37m into the runtook 6m 41s
startup 182.2 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 182.2 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.83 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The canvas zoom changed from 161% to 90%, so the whole view is smaller and shifted. The \"Add Reward Program\" card, the Sam/Owner avatar card and the Alex/Staff popover all sit lower and further left.\n- The right-hand Prototype panel is no ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. The last self-check passed and I submitted at 04:03 UTC, which ends the session.
- What I submitted: the plain LoRA run on Qwen3-VL-8B (run 1), trained on 1,884 pairs, with the language-model layers in FP8, served by vLLM behind the proxy.
- Model choice: run 1, run 3 and a 50/50 weight average of the two scored the same on my dev sets, within judge noise. I kept run 1 because it was already in the output folder and had passed a self-check.
- Self-check: it passed on 250 files, 15.80 GB.
- Startup:
/healthanswered after 182 s, against the 500 s limit. - Throughput: 70 requests at concurrency 14 took 9.83 s (7.1 pairs/s), against the 14 s limit.
- Responses: all 70 parsed and were non-empty. No judge read them.
- Not tested: the check gives no score and ran no architecture review.
- Startup:
- Paperwork:
architecture.jsonandDATA_MANIFEST.mdnow name run 1 as the final model and list the training corpora. The other runs (hint ablation, run 3, the weight average) are described as evaluated but not submitted. I checked that every file they list exists, and found no API keys in the shipped files.