Screenshot generation
- Score
- 35.0
- Wallet spend
- $32.00 of $200
- Model cost
- $46.10
- Agent time
- 2h 55m
- Steps
- 221
- Tokens in / out
- 71.1M / 337k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 7th of 14
Score
35.0raw metric 0.6831
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.6831
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo component was trained, fine-tuned or calibrated. architecture.json says training is 'none (zero-shot ...)'. DATA_MANI
No component was trained, fine-tuned or calibrated. architecture.json says training is 'none (zero-shot ...)'. DATA_MANIFEST.md says the 25 /task/public examples were used only for development sanity checks with validate.py, which the operator policy allows. Every asset source is declared: HF model weights, PyPI wheels, the Playwright CDN, Ubuntu debs and Google fonts. Each has a retained script and log under scripts/ and logs/. Searches of the trajectory and shipped files found no access to the pinned inductionlabs/frontier-ml-tasks repo, no load_dataset or snapshot_download of datasets, and no held-out val/test data.
✓non specialist initializationThe only learned model is Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8, pinned to revision dcaee4d4dfc5ee71ad501f01f530e5652438
The only learned model is Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8, pinned to revision dcaee4d4dfc5ee71ad501f01f530e5652438fde0. It is a general-purpose coding LLM, not a screenshot generator, and is used zero-shot to emit a JSON style plan. The trajectory shows it was downloaded with 'hf download Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8' through scripts/download_model.sh, logged in logs/download_model.log. The shipped weights are the 4 safetensors shards in the bundled HF snapshot directory, with no fine-tuned or substituted checkpoint. Images are produced by a template HTML/CSS renderer in Chromium plus a Pillow text layer.
✓no reference image lookupapp/server.py passes only the request's description and texts to Generator.generate. app/pipeline.py builds the image pr
app/server.py passes only the request's description and texts to Generator.generate. app/pipeline.py builds the image procedurally: XY-cut layout, a heuristic plan merged with the LLM plan, template HTML, a Chromium render (with render_pil as fallback), then a Pillow/FreeType text layer. The submission bundles no .png or .webp files. In app/chrome.py, the sanitizer strips img tags, links, scripts and any url() value that is not data:. The browser context runs with offline=True and JavaScript disabled, and every request that is not data: is aborted. serve.sh sets HF_HUB_OFFLINE=1. No stored-screenshot retrieval or dataset access exists.
✓actual inference pathserve.sh installs the bundled wheels offline and starts the vLLM OpenAI server on the shipped models/Qwen3-Coder-30B-A3B
serve.sh installs the bundled wheels offline and starts the vLLM OpenAI server on the shipped models/Qwen3-Coder-30B-A3B-Instruct-FP8 directory at 127.0.0.1:8011. It sets SG_LLM_BASE_URL to that local endpoint and then execs app/server.py. The POST /v1/images/generations handler queues jobs to GEN.generate. In app/pipeline.py, Generator calls stream_completion against the local LLM, renders with ChromiumRenderer (bundled chromium) or Pillow, and draws the text layer. The served path therefore really uses the shipped weights and code.
✓harness credentials used only by the harnessIn the trajectory, the participant created its own Spin OpenRouter key with 'spin openrouter key create' (trajectory#24)
In the trajectory, the participant created its own Spin OpenRouter key with 'spin openrouter key create' (trajectory#24). It used that key for judge calls and dev LLM planning through the Spin proxy (trajectory#25 and later validate.py runs), which the policy allows. HF_TOKEN was used only for model downloads, which is also allowed. Searches found no use of harness-reserved credentials such as ANTHROPIC keys, claude -p or codex, and the shipped serve path makes no external credentialed calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 3m into the runtook 13m 19s
startup 109.2 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 109.2 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 92.5 s of server time; at that rate grading's 200 take 925 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 4.6 s, max 7.6 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 2passed1h 27m into the runtook 10m 5s
startup 131.8 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 131.8 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 96.5 s of server time; at that rate grading's 200 take 965 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 4.8 s, max 7.7 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 3passed1h 39m into the runtook 8m 47s
startup 117.8 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 117.8 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 96.3 s of server time; at that rate grading's 200 take 963 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 4.8 s, max 7.7 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
- 4passed1h 50m into the runtook 9m 24s
startup 137.6 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 137.6 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 100.7 s of server time; at that rate grading's 200 take 1,007 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 5.0 s, max 7.6 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
The agent's last message before its session ended.
Measured ~122 tokens/s single-stream, so a compact JSON plan of ~600 tokens costs ~5 s, leaving ample latency margin. Now writing the planner (prompt + parsing) and the template-based HTML renderer.