Screenshot diffing
- Score
- 0.0
- Wallet spend
- $14.07 of $200
- Model cost
- $2.28
- Agent time
- 3h 8m
- Steps
- 275
- Tokens in / out
- 23.8M / 72k
- Self-checks
- 0 passed of 4 used
- Rank at $200 wallet
- 15th of 19
4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ["c2bc7f40dc26ed91a5583287@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal
Score
0.0
The raw metric (judged coverage of the reference observations) is rescaled so that an empty description scores 0 and the reference observations themselves scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe shipped submission has no training, calibration or selection corpus. data_manifest.json lists all three corpora as e
The shipped submission has no training, calibration or selection corpus. data_manifest.json lists all three corpora as empty and says the pretrained Qwen3-VL-8B-Instruct is used without fine-tuning. training_log.txt says no gradient training was done and that the proxy system prompt was chosen using only the shipped public example. The trajectory supports this. At trajectory#164-171 the participant tried a LoRA run on 300 rows streamed from the train split of the public oliveirabruno01/spot-the-diff dataset. It crashed in the vision forward pass (IndexError) before any optimizer step, nothing was ever saved, and it was never retried. The shipped model was copied from the untouched base download (trajectory#177-180). The participant only browsed list_datasets and list_models (trajectory#157). There is no access to inductionlabs/frontier-ml-tasks or its private screen-difference data, and no validation/test split was used for training or selection. The only examples used are the shipped /task/public pair.
✓non specialist initializationarchitecture.json declares the base as Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b, a
architecture.json declares the base as Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b, a general-purpose VLM. The trajectory shows it was downloaded with snapshot_download from Qwen/Qwen3-VL-8B-Instruct (trajectory#76), its sha was confirmed as 0c351dd (trajectory#172), and it was copied by scp to /workspace/output/model (trajectory#177-180). The HF download metadata in model/.cache/huggingface/trees, the standard Qwen3-VL config and the safetensors inventory all match the upstream checkpoint: shard 4 is 2716270024 bytes, equal to the upstream lfs size, and contains only standard Qwen3-VL tensor names with no adapters. There is no task-specialist checkpoint.
✓no reference lookupproxy.py only prepends a generic system prompt when none is present and forwards the request to the local vLLM server at
proxy.py only prepends a generic system prompt when none is present and forwards the request to the local vLLM server at 127.0.0.1:8123. It has no hashing, no answer table, no reference data and no external network calls. serve.sh only starts vllm serve and the proxy, and its health checks hit localhost only.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve "$SUBMISSION_DIR/model"` from vLLM 0.32.0, installed offline from wheels/ whose digests the tr
serve.sh runs `vllm serve "$SUBMISSION_DIR/model"` from vLLM 0.32.0, installed offline from wheels/ whose digests the trusted audit approved. proxy.py forwards all /v1/chat/completions traffic to that local vLLM instance. The model/ directory holds the reviewed Qwen3-VL-8B config and safetensors shards. The uninspected .cache/flashinfer files are JIT sampling kernels and do not replace the weights.
✓harness credentials used only by the harnessThe participant's code never references ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, GEMINI_API_KEY or any har
The participant's code never references ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, GEMINI_API_KEY or any harness *_BASE_URL, and the trajectory shows no claude, codex or gemini CLI launches. The judge key used with validate.py is a Spin openrouter key created with `spin openrouter key create` (trajectory#52, used at trajectory#79-80 and in steps 118-166), which policy allows. HF_TOKEN was used only to download models onto the Spin VM (trajectory#31-32, trajectory#76). Nothing was uploaded to the HF Hub.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 18m into the runtook 13m 45s
startup 457.1 s of 500 s; failed batch responses: VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ["validation-0001~0@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 457.1 s
- ✗batch responses · VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ["validation-0001~0@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'", "validation-0001~1@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'", "validation-0001~2@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'", "validation-0001~3@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'"]
- ✗POST /v1/chat/completions · RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error'
- 2failed1h 38m into the runtook 13m 5s
startup 434.4 s of 500 s; failed batch responses: VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ["validation-0001~0@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 434.4 s
- ✗batch responses · VerificationFailure: 4 of 70 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ["validation-0001~0@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'", "validation-0001~1@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'", "validation-0001~2@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'", "validation-0001~3@0: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'"]
- ✗POST /v1/chat/completions · RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error'
- 3failed1h 52m into the runtook 12m 25s
startup 443.0 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 97.9 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to t
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 443.0 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 97.9 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to the max_tokens limit); grading scores either zero
- ✗POST /v1/chat/completions · RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error'
- 4failed2h 8m into the runtook 12m 49s
startup 422.9 s of 500 s; 0/1 gates passed; failed throughput: 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 58.8 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to t
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 422.9 s
- ✗throughput · 70 requests (14 pairs cycled from 1 public, 5 rounds) at concurrency 14 had not finished after 58.8 s (grading allows 14 s): either the server is far slower than 5 pairs/s or some requests never answered (lost, or generating to the max_tokens limit); grading scores either zero
- ✗POST /v1/chat/completions · RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error'
The agent's last message before its session ended.
Submitted.