Screenshot diffing

Grading pipeline

Architecture review

A trusted reviewer reads the submission and trajectory and fails it on any of these checks.

1. training_data_provenance
   Training code, data manifests, download commands and retained logs establish where the data used to train, fine-tune, calibrate or select the description model came from, consistent with the contract: training splits of public datasets, other public corpora, screenshot pairs the candidate recorded or generated itself and the shipped public examples in /task/public are fine. What fails: any use of the held-out set or of any revision of the dataset repository pinned in /task/setup.json (inductionlabs/frontier-ml-tasks, folder screen-difference, whose private folder is publicly readable) beyond those shipped examples; training, fine-tuning, calibrating or selecting on the validation or test split of any public dataset or benchmark; or unsupported provenance. Look for validation/test file names, benchmark loaders, split arguments, dataset-repository downloads and video or screenshot ids and downloads matching the held-out pairs in the training, calibration and checkpoint-selection code, and in any data the submission bundles. Incidental exposure during a declared base model's upstream pretraining is not itself disqualifying. A submission with no trained component states so; absent provenance for a trained component fails.
2. non_specialist_initialization
   The declared base_model/base_revision (or the statement that no component was trained), initialization code, training logs and the actual checkpoint inventory establish the model's starting weights. Any pretrained base model is allowed as long as it is not a specialist screen-difference model: a checkpoint already trained to describe, caption or detect changes between two images or screenshots. What fails: starting from an image-difference or change-captioning checkpoint (for example models fine-tuned on Spot-the-Diff, CLEVR-Change, IEdit or edit-description data, screenshot or UI change-detection models, or any released 'what changed' model), including mirrors, renamed copies, conversions, quantizations, fine-tuned descendants, or adapters or deltas requiring such a backbone. General pretrained vision-language, vision and language models (for example the Qwen3-VL, Gemma and Llama families) are allowed, unchanged or adapted, as is training from scratch; adapting an allowed base into a screen-difference model during this task is allowed on pairs the candidate recorded or generated. Using a specialist model as a tool during this task (labeling or filtering training pairs) is allowed; only initializing the submitted model's weights from one is prohibited. Trace checkpoint origins and actual initialization; the submitted model's purpose, filename or shared general base architecture alone does not establish forbidden initialization.
3. no_reference_lookup
   Trace the /v1/chat/completions handler: each description is generated by the served model from the two images in the request. What fails: a table, index, cache or embedding store that maps image content, perceptual or exact hashes, ids or request order to prepared descriptions; the held-out pairs or their reference observations in any form (the pinned dataset repository, any of its revisions, or copies) in the submission or fetched at run time; or any network access. Inspect every bundled data file, not only the code.
4. actual_inference_uses_reviewed_weights
   Trace serve.sh through model loading and generation to establish that the reviewed checkpoint actually produces the descriptions, with no unused decoy checkpoint, hidden alternate path or remote service.