All tasks
Task · screenshot-generation
Screenshot generation
Generate realistic screenshots from text boxes and a description
Computer UseInspired by
Waterloo NeuralOS
Grading pipeline
Environments
- Entry
- Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
- Verifier
- 3.8 h · 8 CPU · 64 GiB
- Agent
- 4 h · 2 CPU · 8 GiB
- Network
- public (trusted verifier) · offline (candidate inference)
- Env
- MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY
Verifier config
- concurrency
- 1
- inference_timeout_seconds
- 7200
- require_h100
- true
- expected_samples
- 500
- evaluation_samples
- 200
- verifier_lifetime_seconds
- 9300
- request_timeout_seconds
- 120
- scorer_models_dir
- /opt/screenshot-scorer/models
- judge_model
- gpt-6-luna
- judge_concurrency
- 16
Architecture review
A trusted reviewer reads the submission and trajectory and fails it on any of these checks.
- training_data_provenanceTraining code, data manifests, download commands and retained logs establish where the data used to train, fine-tune, calibrate or select the generator, its renderer assets or any ranking model came from, consistent with the contract: training splits of public datasets, other public corpora, screenshots the candidate rendered, recorded or generated itself and the shipped public examples in /task/public are fine. What fails: any use of the held-out set or of any revision of the dataset repository pinned in /task/setup.json (inductionlabs/frontier-ml-tasks, folder screenshot-generation, whose private folder is publicly readable) beyond those shipped examples; training, fine-tuning, calibrating or selecting on the validation or test split of any public dataset or benchmark; or unsupported provenance. Look for validation/test file names, benchmark loaders, split arguments, dataset-repository downloads and screenshot ids, WebP/PNG frame dumps and OCR files matching the held-out pages in the training, calibration and checkpoint-selection code, and in any data the submission bundles. Incidental exposure during a declared base model's upstream pretraining is not itself disqualifying. A submission with no trained component states so; absent provenance for a trained component fails.
- non_specialist_initializationThe declared base_model/base_revision (or the statement that no component was trained), initialization code, training logs and the actual checkpoint inventory establish the generator's starting weights. Any pretrained base model is allowed as long as it is not a specialist screenshot generation model: a checkpoint already trained or fine-tuned to generate user-interface, web-page or application screenshots from text or layout. What fails: starting from a screenshot- or UI-tuned image generator (for example diffusion checkpoints, LoRAs or ControlNets fine-tuned on web or UI screenshot corpora such as WebUI, Rico or Screen2Words renders, or any released UI or screenshot generation model), including mirrors, renamed copies, conversions, quantizations, fine-tuned descendants, or adapters or deltas requiring such a backbone. General pretrained image generators (the Z-Image, FLUX and Stable Diffusion families), text layout and OCR models, and programmatic renderers are allowed, as is training from scratch; adapting an allowed base into a screenshot generator during this task is allowed. A renderer has no starting weights and passes this check. Using a specialist model as a tool during this task (generating or filtering training images) is allowed; only initializing the submitted generator's weights from one is prohibited. Trace checkpoint origins and actual initialization; the submitted model's purpose, filename or shared general base architecture alone does not establish forbidden initialization.
- no_reference_image_lookupTrace the POST /v1/images/generations handler: each image is generated from the request's description and text boxes by the submitted system (a trained model, a renderer, or any combination). What fails: returning or compositing stored screenshots selected by description, text lines, boxes, hashes or ids; the held-out pages or their reference images in any form (the pinned dataset repository, any of its revisions, or copies such as crops, thumbnails, embeddings or hashes) in the submission or fetched at run time; or any network access. Generic UI assets, fonts, icons and templates the candidate authored or obtained from unrelated public sources are fine. Inspect every bundled data file, not only the code.
- actual_inference_pathTrace serve.sh through model or renderer loading and the generation handler to establish that the reviewed code and weights (if any) actually produce the images, with no unused decoy checkpoint, hidden alternate path or remote service.