Screenshot generation
- Score
- 40.6
- Wallet spend
- $14.33 of $200
- Model cost
- $3.66
- Agent time
- 2h 50m
- Steps
- 202
- Tokens in / out
- 51.8M / 311k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 5th of 14
Score
40.6raw metric 0.7064
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.7064
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json says no component was trained, fine-tuned or calibrated, and the trajectory shows no training run. Pip
architecture.json says no component was trained, fine-tuned or calibrated, and the trajectory shows no training run. Pipeline variants were chosen on a dev set built from the participant's own sources. The text lines and boxes come from the hand-written vocabulary and templates in dev_set/make_devset.py, and the descriptions were written by an LLM through the participant's Spin key in dev_set/make_coherent.py. The participant also scored the system on /task/public/validation.jsonl, which is part of the allowed shipped public examples (trajectory#189, #191). I found no download of the pinned inductionlabs dataset repository, no load_dataset calls, and no held-out sample ids in the shipped files.
✓non specialist initializationThe generator is black-forest-labs/FLUX.2-klein-4B, a general-purpose image generator, used without changes. download_mo
The generator is black-forest-labs/FLUX.2-klein-4B, a general-purpose image generator, used without changes. download_models.sh pulls it with snapshot_download at pinned revision e7b7dc27…, and logs/download_models.log records that download. The transformer weight inventory has the standard FLUX.2 layout with no LoRA or adapter tensors. The OCR detector (PP-OCRv5, from ModelScope RapidOCR) is only used to erase stray text and is an allowed OCR model. The trajectory shows no LoRA or other screenshot-specialist checkpoint being used (a general Z-Image base was downloaded only for experiments).
✓no reference image lookupThe POST handler in server.py does the following: parse_request, then render_png, then _background. _background builds a
The POST handler in server.py does the following: parse_request, then render_png, then _background. _background builds a wireframe from the request's own text boxes and runs the FLUX pipeline with a prompt made from the description. textdet.py and inpaint.py then erase stray text, and textrender.py draws the requested text using the bundled OFL fonts. Nothing is looked up from stored screenshots by description, text, hash or id. The only bundled data files are fonts, model weights, wheels and the synthetic dev set. serve.sh turns on offline mode for all model libraries.
✓actual inference pathserve.sh runs server.py with --model-dir models/flux2-klein-4b, --font-dir fonts and --detector-dir models. server.py lo
serve.sh runs server.py with --model-dir models/flux2-klein-4b, --font-dir fonts and --detector-dir models. server.py loads that exact directory through Flux2KleinPipeline.from_pretrained and loads models/ocr/ch_PP-OCRv5_det_server.onnx through the TextDetector in textdet.py. Both are used for every request. I found no decoy checkpoint, alternate path or remote service. The only fallback is a flat background if the model fails to load.
✓harness credentials used only by the harnessThe harness was claude-code, launched with ANTHROPIC_API_KEY. Environment variables appear only in a redacted listing (t
The harness was claude-code, launched with ANTHROPIC_API_KEY. Environment variables appear only in a redacted listing (trajectory#14). The judge and description-LLM calls use a key the participant created with `spin openrouter key create` and sent to the Spin base URL (trajectory#20s, notes in trajectory#101), which is the participant's own allowance and is allowed. I found no step that calls the Anthropic API, copies the harness key, or launches another harness CLI session.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 12m into the runtook 4m 7s
startup 13.6 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 13.6 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 18.2 s of server time; at that rate grading's 200 take 182 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 0.9 s, max 1.1 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 2passed2h 19m into the runtook 5m 14s
startup 15.4 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 15.4 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 69.1 s of server time; at that rate grading's 200 take 691 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 3.5 s, max 3.8 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 3error2h 43m into the runtook 1m 17sfree (stopped before any GPU time)
The self-check could not finish (EvaluationError: Could not fetch the task's public files (ConnectError)).
- 4passed2h 45m into the runtook 5m 24s
startup 15.2 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 15.2 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 69.0 s of server time; at that rate grading's 200 take 690 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 3.4 s, max 3.7 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output (74 files, 16.1 GB) was submitted at 08:49 UTC, and the session is closed.
What it does. serve.sh runs an offline server with POST /v1/images/generations and GET /health. For each request:
- It draws the requested lines into their boxes on a plain background (dark or light, from the description) to make a wireframe.
- FLUX.2-klein-4B (Apache-2.0, pinned commit
e7b7dc27…) edits that wireframe into a full-screen screen that follows the description and keeps the boxes' layout. - The PP-OCRv5 detector (ONNX, run on CPU) finds stray text the model drew outside the boxes, and those pixels are smoothly filled.
- Each requested line is drawn exactly into its box with an OFL UI font, after the box is knocked out.
Every stage falls back to something simpler on error, so every request gets a 1280×720 RGB PNG.
Results
| Set | T | L | F | R | Approx. score |
|---|---|---|---|---|---|
| Validation (reported once, after the design was frozen) | 0.997 | 0.946 | 0.956 | 0.48 | ~0.72 |
| My synthetic dev set (26 screens) | 0.999 | 0.991 | 0.855 | 0.265 | ~0.60 |
The approximate score is (geomean(T,L,F) + R)/2 from the means. On validation, layout recall is the weak part (0.909). Realism depends heavily on text coherence, so my dev set, which has random text, understates R.
Self-checks. Three were run; two passed and the third was an infrastructure error. The final code passed on the H100: startup 15.2 s, p95 3.7 s, no failed requests, all 20 PNGs valid. One self-check remains.
Things you should know
- Validation use. Early on I printed a few validation rows and looked at two example images to understand the format. I did not use validation for any design choice. I ran
validate.pyonce, after the design was frozen, for this report. The architecture review has not run. - Dev set. The synthetic dev set (
dev_set/) is my own. Its descriptions were written by an LLM through spin. Pipeline variants were chosen on it, and no training was done.architecture.jsonsays this and lists the sources. - Realism is a plateau. Raw diffusion backgrounds scored R≈0.23–0.28 whatever the model or prompt. The gains came from layout-aware editing (faithfulness) and stray-text erasing (layout precision).
- Not used: FLUX.2-klein-9B (gated, non-commercial license), and the two-pass, high-resolution and step-count variants (no measurable gain).
- Licenses: FLUX.2-klein-4B (Apache-2.0), PP-OCRv5 detector (Apache-2.0), onnxruntime (MIT), fonts (OFL, with license texts).
- Spend: $14.33 of the $200 spin budget.