Screenshot diffing
- Score
- 8.5
- Wallet spend
- $6.00 of $10
- Model cost
- $8.99
- Agent time
- 2h 16m
- Steps
- 245
- Tokens in / out
- 22.5M / 189k
- Self-checks
- 1 passed of 3 used
- Rank at $10 wallet
- 12th of 19
Score
8.5raw metric 0.08544
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery corpus has a declared source. The first is procedural screenshot pairs the candidate generated itself with generat
Every corpus has a declared source. The first is procedural screenshot pairs the candidate generated itself with generate_synthetic.py: training seeds 0-5999, with development and selection seeds 200000-200255 kept separate from training. The second is ONE-Lab/GUI-World at a pinned revision, and prepare_gui_world.py fetches only the Annotation/train JSONL files and the video_path values listed in them. Those GUI-World pairs were labeled by a general Qwen3-VL-235B teacher called through the Spin OpenRouter proxy, which the policy allows. Model and decoding selection (model-selection.json) used only the candidate's own procedural development seeds. Searches of the code and trajectory found no reference to the pinned inductionlabs/frontier-ml-tasks repository, /task/public, or any validation/test split. The trajectory records the GUI-World download step (trajectory#39).
✓non specialist initializationtrain.py loads Qwen3VLForConditionalGeneration from /workspace/downloads/qwen3-vl-4b. remote_setup.sh downloads that dir
train.py loads Qwen3VLForConditionalGeneration from /workspace/downloads/qwen3-vl-4b. remote_setup.sh downloads that directory with `hf download Qwen/Qwen3-VL-4B-Instruct --revision ebb281ec...`, a general VLM, and trajectory#38 shows the download in progress. Stage one (synthetic-stage/run-config.json) starts with adapter "" on this base. The final run resumes that stage-one adapter and merges it. The shipped tensor inventory (bf16 Qwen3-VL-4B language/vision layers) matches a merged base and shows no specialist change-captioning checkpoint.
✓no reference lookupserver.py forwards the raw request body to a local vLLM backend that serves ROOT/model, using the bundled chat template.
server.py forwards the raw request body to a local vLLM backend that serves ROOT/model, using the bundled chat template. That template only adds a system prompt. response_utils.clean_description only removes duplicate bullets and a truncated trailing bullet from the generated text. There is no table, hash map, cache or embedding store. serve.sh sets HF_HUB_OFFLINE and installs from local wheels with --no-index. Bundled files are the model, wheels, manifests, logs and scripts; none contains held-out pairs or reference descriptions.
✓actual inference uses reviewed weightsserve.sh runs server.py, which starts vllm.entrypoints.openai.api_server with --model ROOT/model, bound to 127.0.0.1. Ev
serve.sh runs server.py, which starts vllm.entrypoints.openai.api_server with --model ROOT/model, bound to 127.0.0.1. Every /v1/chat/completions request goes to that backend. The shipped model/ directory is the scp copy of /workspace/results/final-train/merged (trajectory#135), which matches selected_training_run training/runs/final. There is no alternate model path, decoy checkpoint or remote service.
✓harness credentials used only by the harnessThe teacher-labeling key came from `spin openrouter key create` (trajectory#38). It was written to openrouter_env and us
The teacher-labeling key came from `spin openrouter key create` (trajectory#38). It was written to openrouter_env and used against the Spin gateway, which is the participant's own allowance. The FRONTIER_ROUTER/ANTHROPIC strings appear only in ps output (trajectory#77), and nothing used or copied them afterwards. The trajectory shows no additional claude/codex/gemini sessions and no harness-key use. HF_TOKEN was used only for downloads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 48m into the runtook 5m 39s
startup 157.8 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 20.81 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 157.8 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 20.81 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The design canvas zooms out from 161% to 90%, showing more of the canvas and wider ruler ranges.\n- The design frame moves from approximately (425, 300) to (300, 250) on the canvas, shifting the main content area left and up.\n- The design f ...[clipped]
- ✓healthy after requests
- 2passed2h 2m into the runtook 5m 15s
startup 191.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 191.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 12.75 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The design canvas zooms out from 161% to 90%, showing more of the canvas and wider ruler ranges.\n- The design frame moves from approximately (425, 300) to (300, 250) on the canvas, shifting the main content area left and up.\n- The design f ...[clipped]
- ✓healthy after requests
- 3failed2h 9m into the runtook 6m 55s
startup 176.5 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 56.7 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to t
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 176.5 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 56.7 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to the max_tokens limit); grading scores either zero
- ✗POST /v1/chat/completions · RuntimeError: timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out
The agent's last message before its session ended.
The restored artifact matches the passing configuration and the final audit is clean: all 63 indexed evidence files exist, Python/JSON parse, serve.sh is executable, and there are no symlinks or cache artifacts. I’m submitting it now.