Screenshot diffing
- Score
- 16.0
- Wallet spend
- $6.98 of $10
- Model cost
- $40.79
- Agent time
- 1h 23m
- Steps
- 230
- Tokens in / out
- 26.4M / 181k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 6th of 14
Score
16.0raw metric 0.1596
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is synthetic and generated locally. training/generate_ui.py renders HTML/CSS screens with Playwright
The training corpus is synthetic and generated locally. training/generate_ui.py renders HTML/CSS screens with Playwright, using the commands --count 512 --seed 20261003 and --count 256 --seed 927143 --dense recorded in training/combined_manifest.json. training/combine_data.py merges only the two train.jsonl files. training/train.py reads training/combined_train.jsonl. evidence/training_8b.log reports 704 examples with a data sha256 that matches the manifest. architecture.json and evidence/data_audit.json declare that no public validation or test data was used. In the trajectory, the public example was only viewed (trajectory#7) and later used for the final validator and selfcheck (trajectory#218, trajectory#219), not for training. There are no load_dataset calls and no external dataset downloads.
✓non specialist initializationTraining starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd01ed87e9c1b53cbc748cba10e61
Training starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. training/download_base.py fetches it with snapshot_download, and training/train.py loads it with from_pretrained(local_files_only=True). A fresh LoRA (r16/alpha32, q/k/v/o) is added via get_peft_model. evidence/training_8b.log records this initialization. model/merge_info.json and adapter/adapter_config.json show the adapter merged into model8b with 144 matrices. model/config.json and the shard tensor inventories match the Qwen3-VL-8B architecture. The trajectory shows only Qwen downloads (trajectory/steps-83-104, trajectory/steps-139-155) and the merge command at trajectory/steps-139-155. A rejected 4B experiment also started from a Qwen base.
✓no reference lookupIn server.py, each request decodes the two submitted images and calls preprocess.prepare_views and messages_for. It then
In server.py, each request decodes the two submitted images and calls preprocess.prepare_views and messages_for. It then generates with the vLLM engine at temperature 0. There is no response cache, lookup table, stored descriptions or outbound network call. preprocess.py only computes pixel-difference boxes and crops. runtime_env.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and serve.sh installs wheels offline with --no-index.
✓actual inference uses reviewed weightsserve.sh sources runtime_env.sh and execs server.py. server.py loads MODEL_PATH, defaulting to ROOT/model, into AsyncLLM
serve.sh sources runtime_env.sh and execs server.py. server.py loads MODEL_PATH, defaulting to ROOT/model, into AsyncLLM with FP8 dynamic quantization from the environment. No calibration data or alternate model is used. The shipped model/config.json and safetensors shards are Qwen3-VL-8B tensors, and evidence/artifact_manifest.json hashes show the tested GPU shards are byte-identical to the packaged ones. No other model endpoint is called during inference.
✓harness credentials used only by the harnessIn the trajectory, OPENAI_API_KEY appears only as a name in an environment listing (trajectory#7). The judge base URL ap
In the trajectory, OPENAI_API_KEY appears only as a name in an environment listing (trajectory#7). The judge base URL appears only in validator help output. The participant only viewed help text for the spin openrouter subcommand (trajectory/steps-1-19) and never invoked it. There are no API calls with harness credentials, no copied secrets and no extra harness or judge sessions. curl was used only on local health endpoints. The frontier validation/selfcheck at trajectory#218 and trajectory#219 is ordinary harness usage.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 14m into the runtook 8m 32s
startup 190.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 190.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.67 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The right-hand \u201cPrototype\u201d panel is closed in the AFTER screenshot.\n- The \u201cInteractions\u201d section is no longer visible in the AFTER screenshot.\n- The \u201cScroll behavior\u201d section is no longer visible in the AFTER ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
Trained on 704 original screenshot pairs. Offline H100 self-check passed: 190.1-second startup, 6.56 pairs/second with 14 requests in flight, and 70/70 valid responses.