Screenshot diffing
- Score
- 2.9
- Wallet spend
- $28.33 of $50
- Model cost
- $14.64
- Agent time
- 6h 22m
- Steps
- 347
- Tokens in / out
- 32.3M / 508k
- Self-checks
- 1 passed of 3 used
- Rank at $50 wallet
- 11th of 14
Score
2.9raw metric 0.02857
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTraining used only one corpus: synthetic UI before/after pairs that the participant generated. They were rendered by the
Training used only one corpus: synthetic UI before/after pairs that the participant generated. They were rendered by the shipped code/render.py, code/engine.js and code/templates/*.html with Playwright/Chromium, and the labels come from the scripted change specs. DATA_MANIFEST.md and architecture.json declare this corpus and the render commands. They state that no public dataset was downloaded. code/mktrain.py splits the data with seed 7 into 11,536 training and 400 synthetic dev rows. logs/train.log matches this with 11,536 training rows. The validation set the participant built (/workspace/valdata) was made from its own synthetic pairs (syn-*) (trajectory#226). The single public example, validation-0001, was used only to probe and evaluate the model. It was not used for training. No pinned held-out dataset repo was fetched.
✓non specialist initializationThe base model is Qwen/Qwen3-VL-4B-Instruct at a pinned commit. It is a general-purpose vision-language model, not a spe
The base model is Qwen/Qwen3-VL-4B-Instruct at a pinned commit. It is a general-purpose vision-language model, not a specialised screen-difference model. The session downloaded it from HF with snapshot_download (trajectory#73). code/train_sft.py loads that base and applies LoRA with r=64, and logs/train.log matches. code/merge_lora.py merges the adapter into a dense checkpoint. The post-train pipeline merged into /dev/shm/merged (trajectory#188). That merged checkpoint was packed into model/ (trajectory#250). It is a Qwen3VLForConditionalGeneration, about 8.87 GB of safetensors (trajectory#254).
✓no reference lookupserve.sh only starts vLLM's OpenAI server on the local $PWD/model, with HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE set. Its
serve.sh only starts vLLM's OpenAI server on the local $PWD/model, with HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE set. Its only network call is a localhost /health probe. There is no answer table, no retrieval and no remote call. The final audit grep found no task/public content or held-out references in the bundle (trajectory#344). code/eval_dev.py is a development evaluation tool that only scores synthetic dev data; serving never calls it.
✓actual inference uses reviewed weightsIn every watchdog attempt, serve.sh runs `vllm.entrypoints.openai.api_server --model $PWD/model --served-model-name subm
In every watchdog attempt, serve.sh runs `vllm.entrypoints.openai.api_server --model $PWD/model --served-model-name submission`. model/ holds the merged fine-tuned weights, copied from /dev/shm/merged (trajectory#250). Its size, 8875719408 bytes, matched the merged file (trajectory#296). The participant briefly served the base model for a comparison, then restored the fine-tuned checkpoint (trajectory#294). The packed output model was never replaced. The final self-check served the shipped bundle and passed (trajectory#343).
✓harness credentials used only by the harnessThe judge credentials the participant used came from its own `spin openrouter key create` scoped key, stored in /workspa
The judge credentials the participant used came from its own `spin openrouter key create` scoped key, stored in /workspace/.judge_env (trajectory#151, trajectory#152). code/eval_dev.py reads them from JUDGE_BASE_URL and JUDGE_KEY to evaluate the synthetic dev data. HF_TOKEN was used only to download the base model. The harness's FRONTIER_ROUTER_* variables appear only in a process listing, and nothing shows the participant using them. The secrets scan of the bundle found no keys (trajectory#344).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed5h 6m into the runtook 11m 34s
failed startup: Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ^^^^^^^^^^ (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/ope
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ^^^^^^^^^^ (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 137, in build_async_engine_client_from_engine_args (APIServer pid=187) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=187) ^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 222, in from_vllm_config (APIServer pid=187) return cls( (APIServer pid=187) ^^^^ (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 148, in __init__ (APIServer pid=187) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 124, in make_async_mp_client (APIServer pid=187) return AsyncMPClient(*client_args) (APIServer pid=187) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 835, in __init__ (APIServer pid=187) super().__init__( (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 490, in __init__ (APIServer pid=187) with launch_core_engines(vllm_config, executor_class, log_stats) as ( (APIServer pid=187) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ (APIServer pid=187) next(self.gen) (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 925, in launch_core_engines (APIServer pid=187) wait_for_engine_startup( (APIServer pid=187) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", lin ...[clipped]
- 2failed5h 36m into the runtook 7m
startup 180.6 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 19.27 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 180.6 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 19.27 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- a selection highlight appears over \"0000\" in the background value\n- the design frame card list is dragged from the center of the screen to the middle left\n- the cursor moves from over the top bar to over the layer row profile, and the la ...[clipped]
- ✓healthy after requests
- 3passed6h 8m into the runtook 7m 28s
startup 171.5 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 171.5 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 9.01 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- a selection highlight appears over \"0000\" in the background value\n- the design frame card list is dragged from the center of the screen to the middle left\n- the cursor moves from over the top bar to over the layer row profile, and the la ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Exit 0 — selfcheck #3 PASSED. Getting the full gate margins: