Screenshot diffing
- Score
- 8.2
- Wallet spend
- $33.57 of $200
- Model cost
- $48.88
- Agent time
- 1h 56m
- Steps
- 230
- Tokens in / out
- 31.3M / 226k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 8th of 14
Score
8.2raw metric 0.0823
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancetraining/data_manifest.json declares every training corpus. Each one is original procedural output from shipped generato
training/data_manifest.json declares every training corpus. Each one is original procedural output from shipped generators (generate.py, browser_pairs.py, combine.py), listed with its seed and command, and external_dataset is null for all of them. The corpora are training_corpus with seed 271828 and 1600 examples, plus the stage-2 and stage-3 mixtures built from the varied, browser and desktop sets on their own seeds. In the stage training logs, the example counts and the stage-3 manifest sha match the declared mixtures. Development corpora use disjoint seeds. The public validation file was used only through /task/validate.py and selfcheck. I found no training on it and no download of any external or held-out dataset.
✓non specialist initializationarchitecture.json declares a general-purpose VLM as the base: Qwen/Qwen3-VL-8B-Instruct, pinned to revision 0c351dd01ed8
architecture.json declares a general-purpose VLM as the base: Qwen/Qwen3-VL-8B-Instruct, pinned to revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. The trajectory shows it fetched with training/download_model.py through snapshot_download, using that repo and revision. In training/train.py and training/train_stage2.py, Qwen3VLForConditionalGeneration is loaded from the base directory and LoRA adapters are applied (r16, alpha 32). The adapters are chained stage1→stage2→stage3 and then merged. The initialization events in the training logs show base /workspace/base_model with these adapters. model/config.json (36 layers, hidden 4096, vision depth 27) and the tensor inventory match standard Qwen3-VL-8B. There is no specialist or task-tuned checkpoint.
✓no reference lookupserver.py decodes only the two base64 images in the request. It runs the deterministic pixel-diff views from vision.py (
server.py decodes only the two base64 images in the request. It runs the deterministic pixel-diff views from vision.py (region_boxes, make_views) and builds static prompts with messages_for. It then generates with temperature 0 via vLLM. I found no answer tables, caches, stored descriptions, retrieval or network access. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE and installs only local wheels.
✓actual inference uses reviewed weightsserve.sh runs `exec python server.py`. server.py loads AutoProcessor and a vLLM AsyncLLM, with dynamic FP8 quantization,
serve.sh runs `exec python server.py`. server.py loads AutoProcessor and a vLLM AsyncLLM, with dynamic FP8 quantization, from CHECKPOINT, which defaults to ROOT/model. The model/ directory holds the merged stage-3 weights: 5 safetensors shards and 750 tensors, consistent with evidence/model_manifest.json and the reviewed weight inventory. The trajectory shows the merged stage-3 output moved into output/model. The trusted binary audits show every wheel matches an approved upstream PyPI digest, so I found no altered loading code.
✓harness credentials used only by the harnessThe only model-API credential the participant used was a scoped OpenRouter key they created with `spin openrouter key cr
The only model-API credential the participant used was a scoped OpenRouter key they created with `spin openrouter key create`, which the operator permits. training/judge_observations.py and training/judge_synthetic.py used it to run the Gemini judge on original synthetic development data only. I found no use of harness-reserved credentials such as OPENAI_API_KEY or the harness base URL, and no spawned agent sub-sessions. The BASE_URL matches were only help text.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1error1h 38m into the runtook 6m 2sfree (stopped before any GPU time)
The self-check could not finish (NotFoundError: Modal Sandbox with container ID ta-01M45H6X3W11AZR5RHDJC05Z2S not found. This means this Sandbox has already shut down. (Error code: 6806EYH5)).; No judge read the descriptions: they were checked only for parsing and being non-empty.
1 gates
- ✓review prerequisites
- 2passed1h 45m into the runtook 9m 12s
startup 229.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 229.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 6.65 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The open interactions menu closes, revealing the main content behind it. The left navigation sidebar becomes narrower, and the main canvas expands leftward into the vacated width. The bottom status text changes from \"161%\" to \"90%\"."}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the trained model and serve.sh.
Offline selfcheck passed: 229.9-second startup, 10.53 pairs/second at concurrency 14, and 70/70 valid responses.