Screenshot generation
- Score
- 0.0
- Wallet spend
- $6.28 of $200
- Model cost
- $8.93
- Agent time
- 1h 29m
- Steps
- 178
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 18th of 19
11 of 200 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['test-0367: failed 3 attempts: Submission returned HTTP 500: b\'{"error": "ValueError: y1 must be greater than or equal to y0"}\' | Submission rScore
0.0
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceWhere the data came from is declared and allowed. architecture.json and training/generate_training_data.py describe a sy
Where the data came from is declared and allowed. architecture.json and training/generate_training_data.py describe a synthetic corpus of 625 samples (575 train / 50 internal synthetic holdout) built from procedural templates, and those templates come from the shipped /task/public examples, which are allowed data. The generator contains no /task paths for hidden splits and no dataset loaders. The data_generation.log matches this. In the trajectory, the participant read the public validation data (trajectory#22). The HF search at trajectory#20 only listed datasets; nothing was downloaded from them, and no held-out or test data was obtained.
✓non specialist initializationThe LoRA was trained from the general instruction model Qwen/Qwen2.5-3B-Instruct, pinned at revision aa8e725… in archite
The LoRA was trained from the general instruction model Qwen/Qwen2.5-3B-Instruct, pinned at revision aa8e725… in architecture.json, training/train.log and training/download_models.py. adapter_config.json names that base model, and the adapter inventory contains only rank-16 lora_A/lora_B tensors for the standard projection modules. The tensor shapes match Qwen2.5-3B (hidden size 2048, MLP size 11008, KV size 256). The auxiliary SD-Turbo is a general-purpose public text-to-image model at a pinned revision. Neither starting point is a screenshot-generation specialist.
✓no reference image lookupThe request handler in src/server.py does three things: the LLM produces a scene spec, SD-Turbo generates only the media
The request handler in src/server.py does three things: the LLM produces a scene spec, SD-Turbo generates only the media regions, and src/renderer.py draws the image procedurally with PIL and the bundled fonts. No source file under src/ calls Image.open, reads stored images, or uses network/HTTP clients. The server never retrieves reference screenshots.
✓actual inference pathserve.sh sets offline mode and runs src/server.py. Its load() function loads models/qwen2.5-3b-instruct, applies and mer
serve.sh sets offline mode and runs src/server.py. Its load() function loads models/qwen2.5-3b-instruct, applies and merges models/scene_architect_lora with PeftModel, and loads models/sd-turbo. generate() calls the LLM (_run_llm), then refine_scene_spec, then _run_sd for the media prompt, and finally the renderer. So every shipped weight is actually used to serve /v1/images/generations.
✓harness credentials used only by the harnessThe only credential the trajectory shows the participant creating and using is their own Spin/OpenRouter proxy key (traj
The only credential the trajectory shows the participant creating and using is their own Spin/OpenRouter proxy key (trajectory#35). It served as the judge key for local validation, which is allowed. I found no use of harness-only credentials, and the served code has no credential use or network calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed59m 9s into the runtook 4m 54s
startup 16.8 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 16.8 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 124.6 s of server time; at that rate grading's 200 take 1,246 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 6.2 s, max 6.6 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 2passed1h 23m into the runtook 4m 8s
startup 18.4 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 18.4 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 119.5 s of server time; at that rate grading's 200 take 1,195 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 6.0 s, max 6.3 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests