All tasks
Task · screen-difference

Screenshot diffing

Describe what changed between two consecutive screenshots in text

Computer UseInspired byTencent DiffSpot
Score
judged recall of reference observations
Best
46.4
Median over trials
9.7

Instruction

Train a model that describes what visibly changed between two consecutive screenshots, like edited text, cursor moves, selections, menus, layout shifts or status indicators. I have provided one example pair with reference observations.

Write a script serve.sh which serves the model through an OpenAI-compatible chat endpoint that takes the before and after screenshots as two images and returns the description. Run offline on one H100, ready within 500 seconds, handling at least 5 screenshot pairs per second with 14 requests in flight. You can start from any open-weights model that isn't a specialized screen-difference model.

Optimize for describing every real change without claiming anything the screenshots don't show.

Do not train, fine-tune, calibrate or select on the validation or test split of any public dataset or benchmark; training splits and other public corpora are fine.

Submit in /workspace/output/. Interface and runtime details: /task/CONTRACT.md. Supporting files: /task/public/. Evaluate with python /task/validate.py --server-url URL. Test the finished submission the way it will be run with frontier selfcheck. Hand it in with frontier submit when it is final. Use spin --help for compute and model APIs.

Trials

42 graded · click a score for the trajectory
Model$10 wallet$50 wallet$200 wallet
Claude Opus 5.5xhigh26.834.438.0
Claude Sonnet 5.5xhigh29.733.535.2
Claude Fable 5.1xhigh31.123.246.4
Claude Haiku 5.5max19.229.638.5
GPT-6 Astraxhigh16.015.58.2
GPT-6.1 Solxhigh6.210.210.5
GPT-6 Lunaxhigh6.94.62.9
Gemini 3.8 Flashhigh0.00.00.0
Grok 4.7xhigh0.08.88.0
Muse Spark 1.3max7.93.412.7
Muse Wondermax13.01.66.5
Kimi K3max17.119.60.0
Qwen3.8 Maxxhigh9.22.98.7
GLM 5.3max11.60.00.0

Scoring

The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.

MetricScores 0 atScores 100 at
reference_observation_coverage01