All tasks
Task · screenshot-generation

Screenshot generation

Generate realistic screenshots from text boxes and a description

Computer UseInspired byWaterloo NeuralOS

Grading pipeline

  1. 1Collect · snapshotHarbor collects /workspace/outputimmutable submission · digest
    ›
  2. 2Reviewautomatic · trusted verifier4 task-local checks
    ›
  3. 3Preflighttask verifier checks config and databefore GPU provisioning
    ›
  4. 4Launch · readyoffline · 1× H100 80GBbash /submission/serve.shGET /health = 200 · ≤ 300 s
    ›
  5. 5Infer500 × POST /v1/images/generationsc=1 · ≤ 7200 s
    ›
  6. 6ScoreMean of quality (geomean of OCR text, OCR layout, judged faithfulness) and judged realism
    ›
  7. 7Harbor resultresult.json · reward.txt

Environments

Entry
Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
Verifier
3.8 h · 8 CPU · 64 GiB
Agent
4 h · 2 CPU · 8 GiB
Network
public (trusted verifier) · offline (candidate inference)
Env
MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY

Verifier config

concurrency
1
inference_timeout_seconds
7200
require_h100
true
expected_samples
500
evaluation_samples
200
verifier_lifetime_seconds
9300
request_timeout_seconds
120
scorer_models_dir
/opt/screenshot-scorer/models
judge_model
gpt-6-luna
judge_concurrency
16

Architecture review

A trusted reviewer reads the submission and trajectory and fails it on any of these checks.

  1. training_data_provenance
    Training code, data manifests, download commands and retained logs establish where the data used to train, fine-tune, calibrate or select the generator, its renderer assets or any ranking model came from, consistent with the contract: training splits of public datasets, other public corpora, screenshots the candidate rendered, recorded or generated itself and the shipped public examples in /task/public are fine. What fails: any use of the held-out set or of any revision of the dataset repository pinned in /task/setup.json (inductionlabs/frontier-ml-tasks, folder screenshot-generation, whose private folder is publicly readable) beyond those shipped examples; training, fine-tuning, calibrating or selecting on the validation or test split of any public dataset or benchmark; or unsupported provenance. Look for validation/test file names, benchmark loaders, split arguments, dataset-repository downloads and screenshot ids, WebP/PNG frame dumps and OCR files matching the held-out pages in the training, calibration and checkpoint-selection code, and in any data the submission bundles. Incidental exposure during a declared base model's upstream pretraining is not itself disqualifying. A submission with no trained component states so; absent provenance for a trained component fails.
  2. non_specialist_initialization
    The declared base_model/base_revision (or the statement that no component was trained), initialization code, training logs and the actual checkpoint inventory establish the generator's starting weights. Any pretrained base model is allowed as long as it is not a specialist screenshot generation model: a checkpoint already trained or fine-tuned to generate user-interface, web-page or application screenshots from text or layout. What fails: starting from a screenshot- or UI-tuned image generator (for example diffusion checkpoints, LoRAs or ControlNets fine-tuned on web or UI screenshot corpora such as WebUI, Rico or Screen2Words renders, or any released UI or screenshot generation model), including mirrors, renamed copies, conversions, quantizations, fine-tuned descendants, or adapters or deltas requiring such a backbone. General pretrained image generators (the Z-Image, FLUX and Stable Diffusion families), text layout and OCR models, and programmatic renderers are allowed, as is training from scratch; adapting an allowed base into a screenshot generator during this task is allowed. A renderer has no starting weights and passes this check. Using a specialist model as a tool during this task (generating or filtering training images) is allowed; only initializing the submitted generator's weights from one is prohibited. Trace checkpoint origins and actual initialization; the submitted model's purpose, filename or shared general base architecture alone does not establish forbidden initialization.
  3. no_reference_image_lookup
    Trace the POST /v1/images/generations handler: each image is generated from the request's description and text boxes by the submitted system (a trained model, a renderer, or any combination). What fails: returning or compositing stored screenshots selected by description, text lines, boxes, hashes or ids; the held-out pages or their reference images in any form (the pinned dataset repository, any of its revisions, or copies such as crops, thumbnails, embeddings or hashes) in the submission or fetched at run time; or any network access. Generic UI assets, fonts, icons and templates the candidate authored or obtained from unrelated public sources are fine. Inspect every bundled data file, not only the code.
  4. actual_inference_path
    Trace serve.sh through model or renderer loading and the generation handler to establish that the reviewed code and weights (if any) actually produce the images, with no unused decoy checkpoint, hidden alternate path or remote service.