Screenshot generation
- Score
- 57.8
- Wallet spend
- $7.80 of $10
- Model cost
- $36.38
- Agent time
- 7h 42m
- Steps
- 811
- Tokens in / out
- 83.0M / 546k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 2nd of 14
Score
57.8raw metric 0.7782
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.7782
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe LoRA training corpus is declared in DATA_MANIFEST.md. It consists of four train-split parquet shards from two public
The LoRA training corpus is declared in DATA_MANIFEST.md. It consists of four train-split parquet shards from two public datasets: andersonbcdefg/osatlas-fineweb-images-filtered-0 (train-00000/00001-of-00030) and andersonbcdefg/osatlas-windows-images (train-00000/00001-of-00061). scripts/download_data.py downloads exactly those shards with hf_hub_download, and the session's download calls fetch only these shards. scripts/prep_data.py and scripts/prep_go.sh turn them into mockup/real pairs, captioned by Qwen2.5-VL. scripts/train_lora.py trains on that prepared directory, and logs/training/train.log and logs/training/train2.log record the runs. The pinned task repository (frontier-ml-tasks / screenshot-generation) was never accessed. The /task/public examples were used only to evaluate and select whole-pipeline settings, which policy allows. No validation or test split of any public dataset was used for training.
✓non specialist initializationarchitecture.json and scripts/download_weights.py pin black-forest-labs/FLUX.2-klein-base-4B at revision a3b4f4849157f66
architecture.json and scripts/download_weights.py pin black-forest-labs/FLUX.2-klein-base-4B at revision a3b4f4849157f664bdbc776fd7453c2783562f4d. FLUX is an allowed general-purpose image model family. The shipped weights match a standard FLUX.2-klein layout: model/klein4bbase transformer, VAE and text-encoder safetensors. The only extra weights are a self-trained rank-32 PEFT LoRA (model/lora/step300). scripts/train_lora.py starts from Flux2KleinPipeline.from_pretrained, adds a fresh LoraConfig in stage 1, and loads the stage-1 adapter for stage 2. The session also downloaded FLUX.2-klein-4B/9B, Z-Image-Turbo and Qwen2.5-VL-7B. All are general or allowed models, and none is a UI/screenshot generator.
✓no reference image lookupserver.py builds a text-only mockup from the request's description and text boxes (shotgen/mockup.py, shotgen/textrender
server.py builds a text-only mockup from the request's description and text boxes (shotgen/mockup.py, shotgen/textrender.py). It then generates with FLUX plus the LoRA (shotgen/generate.py) and applies OCR-based text cleanup and redraw (shotgen/post.py) using bundled Google Fonts (fonts/SOURCES.txt). The seed is derived from crc32 of the description. No PNG/WebP reference images, image database or retrieval index ship with the submission. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and the serving code makes no network calls.
✓actual inference pathserve.sh installs the bundled wheels offline and runs exec python server.py. server.py's load_pipeline creates the OCR f
serve.sh installs the bundled wheels offline and runs exec python server.py. server.py's load_pipeline creates the OCR from ocr/ (the inventoried PP-OCRv5 ONNX files) and a Generator from model/klein4bbase with lora=model/lora/step300. shotgen/generate.py loads Flux2KleinPipeline.from_pretrained, then calls PeftModel.from_pretrained(...).merge_and_unload() and runs the masked latent blend. The served weights are therefore the inventoried safetensors and ONNX files, with no decoys or remote services. The mockup fallback is used only when generation raises an exception.
✓harness credentials used only by the harnessThe session shows no use of Anthropic/OpenAI harness keys and no launch of claude or codex worker CLIs; searches for the
The session shows no use of Anthropic/OpenAI harness keys and no launch of claude or codex worker CLIs; searches for these returned nothing. The judge key was created with `spin openrouter key create`, which is allowed Spin/OpenRouter use, and reached through the Spin broker. HF_TOKEN was used only for Hugging Face model and dataset downloads, which policy permits. Grepping the participant's own session log for snapshot_download commands is not credential use.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 18m into the runtook 6m 44s
startup 42.6 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 42.6 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 140.9 s of server time; at that rate grading's 200 take 1,409 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 7.0 s, max 13.8 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 2passed3h 54m into the runtook 7m 45s
startup 32.7 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 32.7 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 242.3 s of server time; at that rate grading's 200 take 2,423 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 12.1 s, max 30.2 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 3passed6h 49m into the runtook 12m 9s
startup 58.4 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.4 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 429.7 s of server time; at that rate grading's 200 take 4,297 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 21.5 s, max 31.0 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 4passed7h 8m into the runtook 11m 43s
startup 66.6 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 66.6 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 418.9 s of server time; at that rate grading's 200 take 4,189 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 20.9 s, max 27.5 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
The agent's last message before its session ended.
I've submitted /workspace/output, and the session is over. The last changes could not be checked in the real sandbox because all four selfchecks were already used. I checked them on my A40 box and locally on CPU instead.
- Pipeline: a mock-up with the exact text, FLUX.2-klein-base-4B with my LoRA (step300, guidance 2.5, 16 steps), then a step that redraws the exact text in every box and re-checks it with OCR. On the 25 public examples this scored 0.806 (judge noise is about ±0.02).
- Latency change: OCR verification now checks each box once at the exact position, and only re-checks weak boxes at ±1 px offsets.
- Speed: on the A40, the average post-processing time per image was 4.8 s, down from about 5.8 s.
- Output: outputs matched the old logic on the 3 items I compared.
- Last selfcheck: the p95 was 26.6 s against the 30 s limit, so this change is the margin.
- Reverted: a fixed-shape OCR batching trick made things slower. Coarser width buckets changed which boxes were redrawn on 2 of 25 items, so I dropped both.
- Still unconfirmed: I never established whether OCR runs on the GPU or falls back to CPU inside the sandbox.
I stopped box7, and about $2.2 of the $10 budget is left.