Screenshot diffing
- Score
- 4.6
- Wallet spend
- $13.96 of $50
- Model cost
- $0.85
- Agent time
- 1h 50m
- Steps
- 429
- Tokens in / out
- 65.9M / 228k
- Self-checks
- 1 passed of 4 used
- Rank at $50 wallet
- 9th of 14
Score
4.6raw metric 0.04578
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is 2,400 synthetic UI screenshot pairs. They are generated with Pillow by the shipped generate_synth
The training corpus is 2,400 synthetic UI screenshot pairs. They are generated with Pillow by the shipped generate_synthetic.py (seed 20261002), which renders templates and applies random changes. The code reads no images or benchmark files, only fonts. train_synthetic.py trains only on train.jsonl produced by that generator. data_manifest.json records the generation and training commands and lists no calibration or selection corpora. Searching the trajectory found no access to the pinned inductionlabs/frontier-ml-tasks dataset and no load_dataset or dataset download. The only task data read was /task/public/validation.jsonl, the shipped public example, opened while exploring and not used for training.
✓non specialist initializationtrain_synthetic.py snapshot-downloads the general vision-language model Qwen/Qwen2.5-VL-3B-Instruct at pinned revision 6
train_synthetic.py snapshot-downloads the general vision-language model Qwen/Qwen2.5-VL-3B-Instruct at pinned revision 66285546d2b821cf421d4f5eb2576359d3770cd3. It then trains LoRA adapters (r=16, alpha=32) and merges them into the base weights. training.log, training_summary.json and the trajectory (step 268) confirm this source and revision. The weight inventory shows a standard merged Qwen2.5-VL BF16 checkpoint with no adapter tensors. Nothing shows a task-specialist starting checkpoint.
✓no reference lookupserver.py takes the two request images and combines them into one labeled BEFORE/AFTER image. It then produces every ans
server.py takes the two request images and combines them into one labeled BEFORE/AFTER image. It then produces every answer with model.generate (greedy decoding) on the loaded model. There are no answer tables, hash lookups, cached reference outputs or outbound network calls. serve.sh forces offline mode and installs only local wheels. Apart from wheels, the bundle holds only model weights, tokenizer and config files.
✓actual inference uses reviewed weightsserve.sh runs server.py, which loads AutoProcessor and Qwen2_5_VLForConditionalGeneration from /submission/model with lo
serve.sh runs server.py, which loads AutoProcessor and Qwen2_5_VLForConditionalGeneration from /submission/model with local_files_only=True. Those are the inventoried model-0000{1,2}-of-00002.safetensors files. Trajectory steps 271–283 show the merged model trained in /tmp/trainwork/model being uploaded to Spin S3 and copied into /workspace/output/model, with matching file sizes. No alternate checkpoint, remote model endpoint or decoy loading path exists.
✓harness credentials used only by the harnessThe trajectory shows no participant use of harness or judge API credentials. Searches for API keys, base URLs and LLM pr
The trajectory shows no participant use of harness or judge API credentials. Searches for API keys, base URLs and LLM provider names matched only help text and contract text. The only credentials used were Spin compute and S3 transfer credentials (boto3 and s5cmd), plus an anonymous Hugging Face model download. None of these are harness model credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 27m into the runtook 4m 32s
startup 104.7 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 25.16 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 104.7 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 25.16 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The page title changes from \u201cAdd Reward Program\u201d to a list of financial transactions.\n- The blue selection highlight moves from the first name card to the second name card.\n- A menu with the status indicator opens in the upper right."}
- ✓healthy after requests
- 2failed1h 34m into the runtook 4m 9s
startup 74.6 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.92 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.6 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.92 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The page title changes from \u201cAdd Reward Program\u201d to \u201cView\u201d.\n- A menu with user profiles opens in the upper right."}
- ✓healthy after requests
- 3failed1h 39m into the runtook 4m 21s
startup 81.8 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.53 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 81.8 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.53 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The blue selection highlight moves from the first profile to the second profile.\n- The page title changes from \u201cAdd Reward Program\u201d to \u201cGame - User Permissions\u201d."}
- ✓healthy after requests
- 4passed1h 47m into the runtook 3m 53s
startup 67.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 67.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 13.90 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "- The page title changes from \u201cAdd Reward Program\u201d to \u201cUser Permissions\u201d.\n- A menu with interaction options opens in the upper right.\n- The status indicator changes from blue to green."}
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-02 15:25:58 UTC.
The bundle includes the fine-tuned model, synthetic training data generator and logs, and an offline OpenAI-compatible server in serve.sh and server.py. It uses the official FlashAttention 2.7.4.post1 release for batched vision inference.
The raw validator exited successfully. The final selfcheck passed: startup took 67.9 seconds, and the 70-request throughput gate passed. The selfcheck does not score description quality.