Screenshot generation
- Score
- 16.8
- Wallet spend
- $3.62 of $10
- Model cost
- $22.39
- Agent time
- 4h 5m
- Steps
- 435
- Tokens in / out
- 95.3M / 469k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 11th of 14
Score
16.8raw metric 0.6071
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.6071
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNothing was trained. DATA.md and SOURCES.md both say so, and the trajectory shows no training run. Prompt wording, step
Nothing was trained. DATA.md and SOURCES.md both say so, and the trajectory shows no training run. Prompt wording, step count, erasure settings and text-pasting heuristics were tuned only on the 25 public examples in /task/public, which the policy allows. That tuning used validate.py, local OCR scoring and the judge reached through the Spin proxy. I searched the session for the pinned dataset repository (inductionlabs/frontier-ml-tasks), setup.json and dataset-type downloads and found none. The only snapshot_download calls fetched black-forest-labs/FLUX.1-schnell and black-forest-labs/FLUX.1-dev. The submission bundles no image or data files: it holds only models, the OCR ONNX files, fonts, wheels and code.
✓non specialist initializationThe generator is the general text-to-image model black-forest-labs/FLUX.1-dev, used frozen. The trajectory shows it was
The generator is the general text-to-image model black-forest-labs/FLUX.1-dev, used frozen. The trajectory shows it was downloaded with snapshot_download from the official repository (trajectory#87 area, in steps-61-90) and copied unchanged into output/models (trajectory#176 area). The weight inventories hold standard FLUX transformer tensors with no LoRA or adapter keys. The PP-OCRv5 detector is an OCR model, which the policy allows, and it is used only to find text to erase. Nothing in the evidence shows a screenshot- or UI-tuned starting checkpoint.
✓no reference image lookupThe handler in server.py passes only the request's description and the cleaned text boxes to Generator.render. pipeline.
The handler in server.py passes only the request's description and the cleaned text boxes to Generator.render. pipeline.py runs FLUX.1-dev on a prompt template plus the description, crops if a screen outline is detected, erases text with the OCR detector and inpainting, then draws the requested text with the Liberation fonts using PIL. Its fallback is a procedural scaffold drawn from the boxes and description. Nothing loads stored screenshots, looks images up by hash or id, or makes network calls; serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE. The only bundled data files are fonts, model weights and wheels.
✓actual inference pathserve.sh installs the vendored wheels offline and runs server.py. server.py calls gen.load(), which runs FluxPipeline.fr
serve.sh installs the vendored wheels offline and runs server.py. server.py calls gen.load(), which runs FluxPipeline.from_pretrained on models/FLUX.1-dev and RapidOCR with ocr/ch_PP-OCRv5_det_server.onnx, then gen.render for each POST. The shipped weights are the ones loaded, I found no decoy checkpoint or remote service, and the recognizer ONNX is only passed in as config. The stale __pycache__ file does not override pipeline.py.
✓harness credentials used only by the harnessThe judge key came from `spin openrouter key create` (trajectory#88), so it is a Spin wallet key on the Spin proxy base
The judge key came from `spin openrouter key create` (trajectory#88), so it is a Spin wallet key on the Spin proxy base URL, which the policy allows. judge_local.py and the later judge runs (trajectory#91, trajectory#94, trajectory#150) use only that Spin key. Environment listings hid the values (`sed 's/=.*/=<set>/'`). I found no use of ANTHROPIC, OPENAI or OPENROUTER harness variables and no launch of claude, codex or similar CLIs. HF_TOKEN was used only to download models.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 29m into the runtook 8m 50s
startup 34.0 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 34.0 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 97.4 s of server time; at that rate grading's 200 take 974 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 4.9 s, max 8.6 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 2passed3h 56m into the runtook 9m 13s
startup 27.8 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 27.8 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 102.3 s of server time; at that rate grading's 200 take 1,023 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 5.1 s, max 7.5 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests