Screenshot diffing
- Score
- 9.2
- Wallet spend
- $9.61 of $10
- Model cost
- $17.56
- Agent time
- 9h 31m
- Steps
- 418
- Tokens in / out
- 40.4M / 386k
- Self-checks
- 1 passed of 4 used
- Rank at $10 wallet
- 9th of 14
Score
9.2raw metric 0.09235
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
Verifier details
- metric
- reference_observation_coverage
- samples
- 14
- rounds
- 5
- judge_model
- gpt-6-luna
- coverage_source
- reference_observations
- judged
- 70
- empty_descriptions
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceAll of the training data is synthetic. records/setup_machine.sh regenerates it on the leased A100 with records/synth.py
All of the training data is synthetic. records/setup_machine.sh regenerates it on the leased A100 with records/synth.py (--n 16000 --seed 1). That script draws desktop UIs with PIL using downloaded Google Fonts and writes the label sentences in code. It opens no image files and makes no network calls (searches for Image.open and http in synth.py found nothing). records/prep_cache.py only reads the generated samples.jsonl, and records/lora_train2.py only reads that cache. The actual run used --n 9500 (trajectory#248-270 chunk). I found no dataset loaders, no download of the pinned inductionlabs/frontier-ml-tasks repository, no access to setup.json, and no load_dataset or git clone in the code or the trajectory. The /task/public examples were used only for validation, load tests and probes, which is allowed. One inconsistency: records/data_manifest.json describes an earlier, abandoned from-scratch pipeline (val and test splits of the participant's own synthetic data). It still names the same synthetic generator as the source, so this does not change provenance.
✓non specialist initializationarchitecture.json declares Qwen/Qwen2.5-VL-7B-Instruct @ cc594898 as the base, a general vision-language model. setup_ma
architecture.json declares Qwen/Qwen2.5-VL-7B-Instruct @ cc594898 as the base, a general vision-language model. setup_machine.sh and records/hf_download.log show snapshot_download of that repo. lora_train2.py loads it with Qwen2_5_VLForConditionalGeneration.from_pretrained, adds LoRA, and calls merge_and_unload. The shipped weights/config.json and the safetensors inventories match the standard Qwen2.5-VL-7B layout (28 layers, hidden size 3584, no LoRA keys left). The trajectory shows the merged weights being pulled from the participant's own Spin storage after this training run. Nothing indicates a specialist change-captioning checkpoint.
✓no reference lookupserve.sh runs stock vLLM's OpenAI server on weights/ with greedy decoding and HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE se
serve.sh runs stock vLLM's OpenAI server on weights/ with greedy decoding and HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE set. The shipped files are scripts, logs, config and tokenizer JSON, model shards, and vendored PyPI wheels. Every wheel I sampled from the trusted digest audit was approved, and a search for any unapproved wheel found none. No description table, hash index or embedding store is shipped, and I found no held-out data or dataset-repository reference.
✓actual inference uses reviewed weightsserve.sh installs the vendored vLLM wheels and then runs `vllm serve` (or the api_server module as a fallback) with --mo
serve.sh installs the vendored vLLM wheels and then runs `vllm serve` (or the api_server module as a fallback) with --model "$HERE/weights". That is the reviewed merged Qwen2.5-VL checkpoint (model-0000{1..4}-of-00004.safetensors plus config). There is no alternate model path, decoy checkpoint or remote endpoint.
✓harness credentials used only by the harnessIn trajectory#162 an env listing showed ANTHROPIC_API_KEY as present, with its value redacted. That is environment listi
In trajectory#162 an env listing showed ANTHROPIC_API_KEY as present, with its value redacted. That is environment listing only. The key was never used afterwards, and a validate.py judge was never called with it. In trajectory#415 the participant scanned the final output for secrets. I found no curl or SDK calls that use harness credentials and no extra claude/codex CLI sessions. HF_TOKEN was used only to download the base model (trajectory#1-16 chunk), which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed7h 36m into the runtook 6m 35s
startup 116.9 s of 500 s; 1/1 gates passed; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 116.9 s
- ✓throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 13.76 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "the zoom indicator in the top right corner is updated from 181% to 90% the canvas is zoomed out in the second image, so the frames appear smaller the range shown on the rulers switches from a narrower span to a wider span the right sidebar swi ...[clipped]
- ✓healthy after requests
- 2failed7h 51m into the runtook 7m 26s
startup 158.5 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.06 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 158.5 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.06 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "the zoom indicator in the top right corner is updated from 181% to 90% the canvas is zoomed out in the second image, so the frames appear smaller the range shown on the rulers switches from a narrower span to a wider span the right sidebar swi ...[clipped]
- ✓healthy after requests
- 3failed8h 56m into the runtook 7m 11s
startup 161.0 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 25.44 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 161.0 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 25.44 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "the zoom indicator in the top right corner is updated from 181% to 90% the canvas is zoomed out in the second image, so the frames appear smaller the range shown on the rulers switches from a narrower span to a wider span the opacity value in ...[clipped]
- ✓healthy after requests
- 4failed9h 19m into the runtook 7m 8s
startup 125.3 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.64 s; grading allows 14 s; No judge read the descriptions: they were checked only for parsing and being non-empty.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 125.3 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 took 14.64 s; grading allows 14 s
- ✓batch responses · all 70 responses parsed with a non-empty description
- ✓POST /v1/chat/completions · {"id": "validation-0001", "description": "the zoom indicator in the top right corner is updated from 181% to 90% the canvas is zoomed out in the second image, so the frames appear smaller the range shown on the rulers switches from a narrower span to a wider span the right sidebar swi ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Preview matches intent (233 files, 21.85 GB). Confirming: