Screenshot generation
- Score
- 21.6
- Wallet spend
- $5.82 of $50
- Model cost
- $5.07
- Agent time
- 1h 53m
- Steps
- 244
- Tokens in / out
- 78.5M / 389k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 7th of 14
Score
21.6raw metric 0.6268
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.6268
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing in the submission is trained. architecture.json lists no training evidence files and declares base_model none /
Nothing in the submission is trained. architecture.json lists no training evidence files and declares base_model none / from-scratch. The only data used to make choices was a synthetic development set. dev/make_dev_set2.py generates it from descriptions and strings written into the script, and it does not read any dataset. The participant also read /task/public/validation.jsonl, the 25 public examples shipped with the task, which the contract explicitly allows. They used it for summary statistics and for report-only validate.py runs (trajectory#12-25, trajectory#185-203). The trajectory has no download of the pinned frontier-ml-tasks repository, no access to its private folder, and no use of another benchmark's validation or test split. The FLUX.1-schnell experiment was dropped and no trace of it is shipped.
✓non specialist initializationThe shipped generator is a programmatic renderer made of scene.py, textlayout.py and renderer.py, which draw with PIL an
The shipped generator is a programmatic renderer made of scene.py, textlayout.py and renderer.py, which draw with PIL and numpy. It has no starting weights. No checkpoint files are shipped, only code and open-licence fonts. Earlier in the session the participant tried a general FLUX.1-schnell diffusion model, which is allowed, then removed it (dev/DEVELOPMENT_LOG.md section 1). None of it is in the final submission.
✓no reference image lookupserver.py checks the request and calls ScreenRenderer.render(description, texts). renderer.py calls scene.build_scene, w
server.py checks the request and calls ScreenRenderer.render(description, texts). renderer.py calls scene.build_scene, which draws shapes procedurally from regex keyword hints in the description and from the text-box geometry. textlayout.draw_texts then draws the requested strings with the bundled fonts. The serving code makes no network calls and opens no files other than the fonts in fonts/. No stored screenshots, images, hashes or held-out data are bundled.
✓actual inference pathThe call chain is serve.sh, then server.py (load_service builds ScreenRenderer and runs a warm-up), then renderer.render
The call chain is serve.sh, then server.py (load_service builds ScreenRenderer and runs a warm-up), then renderer.render, then scene.build_scene and textlayout.draw_texts, and finally encode_png, whose output is returned as b64_json. No decoy checkpoint, alternate path or remote service exists. The submission ships no weights, and the earlier FLUX background path was removed from server.py (trajectory#40-60).
✓harness credentials used only by the harnessThe judge calls went through a key the participant created with `spin openrouter key create` (trajectory#27, trajectory#
The judge calls went through a key the participant created with `spin openrouter key create` (trajectory#27, trajectory#28). That is the participant's own Spin allowance. They stored it in /tmp/judge_env.sh, outside the submission, and dev/score_dev.py uses it through MLE_JUDGE_API_KEY/JUDGE_BASE. The trajectory has no use or copying of ANTHROPIC or other harness credentials or base URLs, and no extra claude, codex or similar CLI sessions. A `claude` entry that appears in trajectory#54 is only a directory name in an `ls ~/.cache` listing.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 40m into the runtook 1m 1s
startup 1.2 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 1.2 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 6.6 s of server time; at that rate grading's 200 take 66 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 0.3 s, max 0.7 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 2passed1h 43m into the runtook 1m 29s
startup 2.5 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 2.5 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 6.7 s of server time; at that rate grading's 200 take 67 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 0.3 s, max 0.7 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 3passed1h 48m into the runtook 55s
startup 0.7 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 0.7 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 6.6 s of server time; at that rate grading's 200 take 66 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 0.3 s, max 0.8 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 4passed1h 53m into the runtook 1m 31s
startup 2.2 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 2.2 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 6.4 s of server time; at that rate grading's 200 take 64 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 0.3 s, max 0.7 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output (24 files, 3.5 MB) was submitted for grading. Its final self-check passed, but the score is still unknown.
What I built: a CPU-only procedural renderer, not a diffusion model. serve.sh starts a server with POST /v1/images/generations and GET /health. The renderer reads the description for theme, accent colour, window style, and named regions. It builds UI structure from the text boxes: bars, sidebars, cards, table rules, buttons, icons. Then it draws each requested string inside its box with an open-licence font. Nothing else is written. No model weights are shipped, so architecture.json lists base model none.
Why I dropped FLUX: I first built FLUX.1-schnell backgrounds with exact text drawn on top. Its layouts scored realism ≈ 0.15–0.22 and faithfulness ≈ 0.36 from the judge. Clean FLUX backgrounds with no text still scored about 2/9 on realism, and cleanup, quantising, and img2img didn't help. Clean procedural UIs scored 5–6/9 on the same probes.
Measured results (judge is gpt-6-luna; text and layout use the pinned PP-OCRv5 scorer on CPU):
| Set | T | L | F | R |
|---|---|---|---|---|
| Synthetic dev (48 samples) | 0.995 | 0.996 | 0.74 | 0.57 |
| Public validation (25 samples, reporting only) | 0.993 | 0.967 | 0.53 | 0.50 |
The public-validation score works out to roughly 0.65. Realism and faithfulness are the weak terms, and the judge's run-to-run noise is about ±0.03.
Serving checks: all four frontier selfcheck runs were used. The last one passed on the exact submitted folder: startup 2.2 s, zero failed requests, p95 0.52 s, and 20 of 20 valid responses. The grader's own run isn't reported to me.
Rules and provenance:
- The validation split was used only to view two example images and read one description, to compute summary stats, and for two
validate.pyreports. No parameter was selected on its scores. - Description keyword rules were added after the first validation report (F 0.52, R 0.46). They were motivated by general UI vocabulary and checked only on synthetic sets. I'm flagging this in case you want to judge that call yourself.
- The development corpus is a synthetic set from
dev/make_dev_set2.py. Its download and generation steps are documented inarchitecture.json. - Fonts are from Google Fonts and DejaVu, under OFL and the DejaVu licence.
Two costs: about $13 of the $50 Spin budget was used, and the architecture review isn't run until grading.