Screenshot diffing
- Score
- 0.0
- Wallet spend
- $144.70 of $200
- Model cost
- $10.71
- Agent time
- 7h 43m
- Steps
- 566
- Tokens in / out
- 51.8M / 302k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 14
Score
0.0raw metric 0
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is declared in data/DATA_MANIFEST.md as the train split of PatronusAI/figmatrace. The download patte
The training corpus is declared in data/DATA_MANIFEST.md as the train split of PatronusAI/figmatrace. The download pattern is data/train-*.parquet, and 26,600 of 158,367 consecutive-state pairs were sampled with seed 17. data/extract_pairs.py and data/build_dataset.py only read the train shards passed to them. Teacher captions in data/captions_luna.jsonl come from openai/gpt-5.6-luna via the allowed Spin gateway. The selection/dev set (data/dev_indices.json, with references from data/dev_references.py) is also drawn from train-split sessions. The trajectory confirms train-shard-only downloads (trajectory#18, trajectory#126). There is no sign of any download of the test/validation split or of the inductionlabs/frontier-ml-tasks screen-difference folder.
✓non specialist initializationtraining/train_sft.py loads general-purpose Qwen2_5_VLForConditionalGeneration weights from a local copy of Qwen/Qwen2.5
training/train_sft.py loads general-purpose Qwen2_5_VLForConditionalGeneration weights from a local copy of Qwen/Qwen2.5-VL-3B-Instruct, pinned to a revision in architecture.json. It then does full SFT with the vision tower frozen. The tensor inventory (hidden 2048, MLP 11008, KV 256 with q/k/v biases) matches the Qwen2.5-VL-3B architecture, and model/config.json agrees. A FigmaTrace-specialized SFT model was mentioned in the session (trajectory#16) but never downloaded or used.
✓no reference lookupserve.sh launches only the vLLM OpenAI server on ./model, installs offline from ./wheels, and makes no network calls. Th
serve.sh launches only the vLLM OpenAI server on ./model, installs offline from ./wheels, and makes no network calls. The only custom serving code is rep_penalty.py's CaptionGuard logits processor. It applies a count-based repetition penalty, a 4-line / 224-token cap and forced EOS, using a fixed set of newline token IDs. It contains no caption tables, image hashes, reference lookups or external calls.
✓actual inference uses reviewed weightsserve.sh runs `vllm.entrypoints.openai.api_server --model $PWD/model --served-model-name submission` with no alternate m
serve.sh runs `vllm.entrypoints.openai.api_server --model $PWD/model --served-model-name submission` with no alternate model path or proxy. The served weights are the reviewed model/model-00001-of-00002.safetensors and model/model-00002-of-00002.safetensors shards, whose inventories match a Qwen2.5-VL-3B checkpoint. rep_penalty.py only adjusts logits from that model.
✓harness credentials used only by the harnessThe participant created their own Spin OpenRouter key (`spin openrouter key create`, trajectory#36). They used it as SPI
The participant created their own Spin OpenRouter key (`spin openrouter key create`, trajectory#36). They used it as SPIN_OR_KEY for teacher captioning, dev references and judging via the Spin gateway, which the policy allows (see data/teacher_call.py, data/caption_daemon.py, data/dev_references.py). HF_TOKEN was used only for dataset/model downloads. No use of harness-reserved credentials was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed7h 34m into the runtook 9m 44s
startup 251.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 251.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 11.94 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The canvas zoom changed from 104% to 52%, as shown in the upper-right zoom control.\nThe canvas view was panned and repositioned: the main dashboard frame moved from approximately x396\u2013870, y150\u2013500 to x400\u2013866, y150\u2013495, w ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Selfcheck PASSED — startup 251s (limit 500), throughput 5.86 pairs/s (70 requests in 11.94s vs 14s limit), all 70 descriptions parsed. Submitting: