Screenshot generation
- Score
- 64.6
- Wallet spend
- $58.14 of $200
- Model cost
- $44.26
- Agent time
- 10h 35m
- Steps
- 936
- Tokens in / out
- 98.6M / 594k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 2nd of 14
Score
64.6raw metric 0.8063
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.8063
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data is declared in architecture.json and DATA_MANIFEST.md, and the code is in data_scripts/harvest_wildgui
The training data is declared in architecture.json and DATA_MANIFEST.md, and the code is in data_scripts/harvest_wildgui.py. It is frames from the public HF dataset joker-112/WildGUI_Screenshots, taken from tar shards part16-19 (3597 rows). The harvest metadata records the shard and member for each frame. The dataset README shows the dataset has no validation/test split, and the dev video ids were kept out of training. The /task/public examples were used only in allowed ways: smoke tests, latency checks and reporting. I found no access to the pinned task repo. Captions were written by a VLM through the Spin proxy, which is a declared use.
✓non specialist initializationtrain_lora.py loads the general base model Qwen/Qwen-Image-2.1, whose download is recorded in logs/download_qwen_image_2
train_lora.py loads the general base model Qwen/Qwen-Image-2.1, whose download is recorded in logs/download_qwen_image_2_1.log. It attaches a freshly initialized peft LoRA (r=64, init_lora_weights=True) and only ever resumes from its own checkpoints. trajectory#923 shows the shipped lora/lora.safetensors has the same md5 (2901e971...) as run1/ckpt_4672, which matches lora/meta.json at step 4672. Nothing shows initialization from a specialist screenshot or UI model.
✓no reference image lookupserver.py passes the request's description and text boxes to screenshot.Generator. Generator builds a layout skeleton, r
server.py passes the request's description and text boxes to screenshot.Generator. Generator builds a layout skeleton, runs Qwen-Image diffusion with the merged LoRA, then runs finisher.py. finisher.py uses PP-OCR to erase generated text and re-render the requested strings. fallback.py draws a flat canvas. None of these read stored screenshots or retrieve anything from disk or the network, and the submission ships no reference images. serve.sh sets HF_HUB_OFFLINE=1.
✓actual inference pathThe serving chain is serve.sh → exec python server.py → screenshot.Generator(weights/Qwen-Image-2.1, ocr_models, lora/lo
The serving chain is serve.sh → exec python server.py → screenshot.Generator(weights/Qwen-Image-2.1, ocr_models, lora/lora.safetensors). Generator merges the shipped LoRA into the shipped transformer weights and runs diffusion on every request. The rendered fallback is used only on timeout or error. I found no decoy weights or alternate serving path.
✓harness credentials used only by the harnessThe trajectory shows no use of ANTHROPIC/OPENAI/BASE_URL harness credentials and no claude or codex CLI launches. Captio
The trajectory shows no use of ANTHROPIC/OPENAI/BASE_URL harness credentials and no claude or codex CLI launches. Captioning calls went through the separately provided Spin openrouter key, read from /workspace/scratch/.judge.json in data_scripts/caption_frames.py (trajectory#94, trajectory#96). That is not the harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 33m into the runtook 3m 58sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed2h 41m into the runtook 5m 42s
startup 24.5 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 24.5 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 5.2 s of server time; at that rate grading's 200 take 52 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 0.3 s, max 0.5 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 3passed3h 16m into the runtook 9m 28s
startup 52.7 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 52.7 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 163.4 s of server time; at that rate grading's 200 take 1,634 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 8.2 s, max 10.3 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 4passed8h 2m into the runtook 12m 53s
startup 68.2 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.2 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 365.7 s of server time; at that rate grading's 200 take 3,657 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 18.3 s, max 24.1 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 5passed10h 22m into the runtook 13m 31s
startup 75.8 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.8 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 384.4 s of server time; at that rate grading's 200 take 3,844 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 19.2 s, max 24.0 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
The agent's last message before its session ended.
The submission is handed in and the session is over. The last frontier selfcheck passed all gates.
| Gate | Result | Limit |
|---|---|---|
| Startup | 75.8 s | 300 s |
| p95 latency | 23.4 s (mean 19.2 s, max 24.0 s) | 30 s |
| Failed requests | 0 of 20 | under 5% |
| Valid images | 20 of 20 | 20 |
| Projected time for 200 requests | about 3,844 s | 7,200 s |
The selfcheck only sends public inputs and doesn't score anything. The architecture review did not run, and the self-check notes say it can still score the submission zero.
I kept the first LoRA training run (step 4672) as the shipped adapter. The second run didn't improve it: mean realism rating on the dev frames was 0.605 against 0.622. It is documented in DATA_MANIFEST.md as evaluated but not shipped.
Final changes this segment:
- Border/faint-text fix: a new finisher pass redraws lines the OCR detector doesn't cover, up to 2 rounds. On the dev frames it left realism unchanged (0.668 and 0.577 against 0.667 and 0.575).
- Large requests: requests with more than 150 boxes now verify with a single crop offset. This brought the 400-box stress case from 28 s to 20 s of finishing time.
- Docs and logs:
architecture.jsonandDATA_MANIFEST.mdnow list the new dev-eval and run-2 logs.
Held-out requests with more boxes will run slower than the public ones. A 400-box request would still take roughly 27 s end to end, by my estimate.