Screenshot diffing
- Score
- 8.0
- Wallet spend
- $13.97 of $200
- Model cost
- $34.78
- Agent time
- 2h 31m
- Steps
- 254
- Tokens in / out
- 59.3M / 98k
- Self-checks
- 1 passed of 3 used
- Rank at $200 wallet
- 9th of 14
Score
8.0raw metric 0.07964
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo weights were trained. The only selection step was prompt wording, and its sources are declared. train.py writes three
No weights were trained. The only selection step was prompt wording, and its sources are declared. train.py writes three synthetic change notes that are authored in the file, and training_log.jsonl records them with weight_update false. data_manifest.json declares the shipped /task/public example as the only other source. The trajectory confirms this. The participant viewed /task/public/media/000000.png and 000001.png (trajectory#81-104) and judged prompt variants against that one public pair's five observations using a Spin OpenRouter key (trajectory#105, trajectory#112). I found no dataset-repository (inductionlabs/frontier-ml-tasks) downloads, no access to the private held-out folder and no use of validation or test splits. Tuning on the shipped public example is allowed. The server.py SYSTEM prompt is heavily fitted to that example, but this is allowed by the policy.
✓non specialist initializationarchitecture.json declares the base as Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0 with no weight update. download_mo
architecture.json declares the base as Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0 with no weight update. download_model.sh fetches the four safetensors shards straight from that Hugging Face revision. The weight inventories for model/model-0000{1..4}-of-00004.safetensors show the standard Qwen3-VL-8B tensor layout (language_model layers, model.visual blocks, lm_head 151936x4096). Nothing shows a change-captioning or screen-difference checkpoint, adapter or delta. Qwen3-VL is an allowed general vision-language base.
✓no reference lookupIn server.py, the /v1/chat/completions handler decodes the two data-URL images and builds a chat prompt from a fixed sys
In server.py, the /v1/chat/completions handler decodes the two data-URL images and builds a chat prompt from a fixed system prompt and the user text. It then sends the images and prompt to the vLLM engine.generate call with temperature 0. There is no table, hash index, cache of descriptions or network access. The bundled files are model configs and weights, wheels, and vLLM/FlashInfer compile caches, none of which hold reference observations. The fixed prompt mentions features of the public example but contains no held-out data and does not stand in for model generation.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and then runs server.py. server.py's load_engine builds vllm.LLM(model=MOD
serve.sh installs the vendored wheels offline and then runs server.py. server.py's load_engine builds vllm.LLM(model=MODEL_DIR), where MODEL_DIR defaults to ROOT/model, the directory holding the inventoried safetensors shards. The served text comes only from that engine's generate output. There is no alternate checkpoint or remote service. The vllm wheel is uninspected, but it is an ordinary inference-engine dependency and no evidence shows it doing anything forbidden.
✓harness credentials used only by the harnessSearches of the trajectory found no use of harness API keys or base URLs (no *_API_KEY or *_BASE_URL references) and no
Searches of the trajectory found no use of harness API keys or base URLs (no *_API_KEY or *_BASE_URL references) and no launch of claude, codex or other agent CLIs. The participant's own model calls used a key from `spin openrouter key create` against api.spin.inductionlabs.com (trajectory#105, trajectory#112), which is the allowed Spin allowance.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 41m into the runtook 13m 11s
failed startup: Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: load_engine() File "/submission/server.py", line 178, in load_engine _tolerate_torchcodec()
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: load_engine() File "/submission/server.py", line 178, in load_engine _tolerate_torchcodec() File "/submission/server.py", line 164, in _tolerate_torchcodec import vllm.multimodal.media.audio as audio File "/opt/venv/lib/python3.12/site-packages/vllm/multimodal/__init__.py", line 3, in <module> from .hasher import MultiModalHasher File "/opt/venv/lib/python3.12/site-packages/vllm/multimodal/hasher.py", line 17, in <module> from .media import MediaWithBytes File "/opt/venv/lib/python3.12/site-packages/vllm/multimodal/media/__init__.py", line 3, in <module> from .audio import AudioEmbeddingMediaIO, AudioMediaIO File "/opt/venv/lib/python3.12/site-packages/vllm/multimodal/media/audio.py", line 40, in <module> from torchcodec.decoders import AudioDecoder File "/opt/venv/lib/python3.12/site-packages/torchcodec/__init__.py", line 13, in <module> from . import decoders, encoders, samplers, transforms # noqa ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/torchcodec/decoders/__init__.py", line 7, in <module> from .._core import AudioStreamMetadata, VideoStreamMetadata File "/opt/venv/lib/python3.12/site-packages/torchcodec/_core/__init__.py", line 8, in <module> from ._metadata import ( File "/opt/venv/lib/python3.12/site-packages/torchcodec/_core/_metadata.py", line 16, in <module> from torchcodec._core.ops import ( File "/opt/venv/lib/python3.12/site-packages/torchcodec/_core/ops.py", line 35, in <module> load_image_library() File "/opt/venv/lib/python3.12/site-packages/torchcodec/_internally_replaced_utils.py", line 71, in load_image_library torch.ops.load_library(image_library_path) File "/opt/venv/lib/python3.12/site-packages/torch/_ops.py", line 1505, ...[clipped]
- 2failed1h 56m into the runtook 13m 55s
failed startup: Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: s/stable/distributed.html#shutdown (function operator()) (MainProcess pid=186) INFO 10-09 08:56:14
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 500s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: s/stable/distributed.html#shutdown (function operator()) (MainProcess pid=186) INFO 10-09 08:56:14 [utils.py:638] [shutdown] Process manager: send sigterm to process EngineCore Traceback (most recent call last): File "/submission/server.py", line 316, in <module> main() File "/submission/server.py", line 311, in main load_engine() File "/submission/server.py", line 213, in load_engine engine = LLM( ^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/llm.py", line 344, in __init__ self.llm_engine = LLMEngine.from_engine_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 185, in from_engine_args return cls( ^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/llm_engine.py", line 111, in __init__ self.engine_core = EngineCoreClient.make_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 126, in make_client return SyncMPClient( ^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 917, in __init__ super().__init__( File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 674, in __init__ with launch_core_engines( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1268, in launch_core_engines wait_for_engine_startup( File "/opt/venv/lib/python3.12/site ...[clipped]
- 3passed2h 18m into the runtook 9m 27s
startup 201.3 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 201.3 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.12 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The canvas is zoomed out from 161% to 90%. The cursor moved from the main dashboard area to the sidebar, which now has a blue outline. The right sidebar changed from an \"On click\" interactions sidebar to a \"Prototype\" sidebar. The top and ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The screenshot-difference server is submitted. The official check passed before hand-in.
It serves Qwen3-VL-8B-Instruct through an OpenAI-compatible chat endpoint. Each description is generated from the two screenshots in the request. On the public example it reports the canvas zoom from 161% to 90%, the cursor moving from the main dashboard area onto a sidebar with a blue outline, the right sidebar changing from an On click interactions panel to a Prototype sidebar, and wider top and left rulers. A text judge marked all five of those observations supported.
The same check measured health in 201 seconds of the 500-second limit, and 8.6 pairs per second against the required 5, with all 70 concurrent responses returning a description.