Screenshot diffing
- Score
- 5.3
- Wallet spend
- $8.24 of $10
- Model cost
- $1.64
- Agent time
- 7h 40m
- Steps
- 664
- Tokens in / out
- 64.1M / 485k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 17th of 19
Score
5.3raw metric 0.05342
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓training data provenancetraining/data_manifest.json declares every corpus, each with a retained download command: FigmaTrace train shards 00000-
training/data_manifest.json declares every corpus, each with a retained download command: FigmaTrace train shards 00000-00011 only, cursor pairs taken from the same train split, and teacher labels from qwen3-vl-30b-a3b-instruct through the Spin API. training/prepare_figmatrace_pairs.py reads only train-*.parquet. The download logs and trajectory steps 26-48 and 132-161 show that only train shards were fetched. The session listed the repo's test shards but never downloaded them. The evaluation and selection pairs also come from the train split. training/label_pairs.py does the teacher labeling with a declared Spin endpoint.
✓non specialist initializationarchitecture.json declares Qwen/Qwen2.5-VL-3B-Instruct at a pinned revision with specialized_screen_difference_model=fal
architecture.json declares Qwen/Qwen2.5-VL-3B-Instruct at a pinned revision with specialized_screen_difference_model=false. training/train_lora.py loads that general base with Qwen2_5_VLForConditionalGeneration.from_pretrained and applies a peft LoRA (r32, alpha 64). training/merge_lora.py merges the adapter with merge_and_unload. In the trajectory the base comes from a snapshot_download of the pinned Qwen revision, and the final model was merged from that base plus the session-trained adapter (adapter-cont-final). model/config.json and the safetensors inventories show standard Qwen2.5-VL tensors (36 layers, hidden size 2048). A Figmatrace-SFT model came up during web research but was never downloaded or used.
✓no reference lookupserve.sh only installs vendored wheels offline with HF_HUB_OFFLINE=1 and starts the stock vLLM OpenAI server on ./model.
serve.sh only installs vendored wheels offline with HF_HUB_OFFLINE=1 and starts the stock vLLM OpenAI server on ./model. There is no custom handler, no network access and no bundled reference or answer table. The checkpoint holds only standard model tensors and a patched chat template. The trajectory shows /task/public being inspected but no lookup of hidden references being built into serving.
✓actual inference uses reviewed weightsserve.sh runs `vllm.entrypoints.cli.main serve "$HERE/model" --served-model-name submission`, so serving loads the shipp
serve.sh runs `vllm.entrypoints.cli.main serve "$HERE/model" --served-model-name submission`, so serving loads the shipped model/ directory. That directory's config.json and two safetensors shards were inspected and match the declared merged Qwen2.5-VL-3B architecture. Every vendored wheel is approved by the trusted binary audits, and no alternate model path or remote backend exists.
✓harness credentials used only by the harnessIn the trajectory, ANTHROPIC_API_KEY shows up only as a redacted name in an `env | grep` listing (trajectory#119). No ca
In the trajectory, ANTHROPIC_API_KEY shows up only as a redacted name in an `env | grep` listing (trajectory#119). No calls to api.anthropic.com or `claude -p` were found. The Spin key the participant used for teacher labeling (trajectory#71, trajectory#121, training/label_pairs.py) is a participant-provided Spin wallet credential for a declared data-generation service, not a harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed16m 36s into the runtook 1m 55sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed18m 45s into the runtook 9m 33s
startup 171.1 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 171.1 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.93 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The visible changes between the two screenshots are:\n\n1. The \"Add Reward Program\" button has been removed from the interface.\n2. The \"Stamps Cards\" section under \"REWARDS PROGRAMS\" has been added to the interface.\n3. The \"Messages\" ...[clipped]
- ✓healthy after requests
- 3passed3h 59m into the runtook 5m 43s
startup 184.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 184.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 10.48 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The \"Add Reward Program\" button is no longer visible in the second screenshot.\nThe \"Sam\" profile card is now selected in the second screenshot, indicated by a blue border around it.\nThe \"Alex\" profile card is now selected in the second ...[clipped]
- ✓healthy after requests
- 4passed7h 20m into the runtook 5m 20s
startup 175.4 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 175.4 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 8.14 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The \"Add Reward Program\" button is no longer visible in the second screenshot.\nThe \"Messages\" section with a notification count of \"5\" is now visible in the second screenshot.\nThe \"Team Members\" section is now visible in the second s ...[clipped]
- ✓healthy after requests
- 5passed7h 31m into the runtook 4m 45s
startup 151.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 151.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 7.35 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "The \"Add Reward Program\" button is no longer visible in the second screenshot.\nThe \"Messages\" section with a notification count of \"5\" is now visible in the second screenshot.\nThe \"Team Members\" section is now visible in the second s ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output (195 files, 12.48 GB):
Model — Qwen2.5-VL-3B-Instruct (not a screen-difference model) with a rank-32 LoRA adapter merged into the weights, in two phases: 626 steps on 5,321 teacher-labelled FigmaTrace pairs, then 394 more steps on 2,004 pairs that were labelled afterwards plus a 1,196-pair replay. Everything comes from the train split of PatronusAI/figmatrace (+ synthetic cursor pairs) with observations written by qwen3-vl-30b-a3b-instruct through the Spin endpoint; no validation/test split of anything, and the public example pair was only read, never trained or selected on.
Checkpoint choice — the gpt-6-luna judge was down for the second phase, so the checkpoints were compared judge-free on 200 freshly labelled pairs (804 reference observations, never trained on): the shipped continuation is better on 42 pairs, worse on 21, equal on 137 (sign test p = 0.011), mentioning 176/804 observations vs 142 for the previous checkpoint, and the gain is semantic (it names the change the reference describes where the earlier one named a different change). The real-judge sweep on the earlier phase is in the logs (base 0.10 → 0.20–0.27 for the adapters), with the caveat that 48 pairs carry ≈0.05 standard error.
Serving — serve.sh installs 157 vendored wheels offline into /tmp and runs vLLM with --max-num-batched-tokens 32768, which is what keeps the vision encoder from re-encoding the screenshots every round.
Verification — final frontier selfcheck of the submitted bytes on an H100: startup 151.9 s (limit 500), throughput 9.52 pairs/s (limit 5), all 70 graded-shape responses parsed and non-empty. The A40 lease was released.