Screenshot generation
- Score
- 0.0
- Wallet spend
- $5.24 of $50
- Model cost
- $11.49
- Agent time
- 2h 44m
- Steps
- 270
- Tokens in / out
- 59.5M / 277k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 11th of 14
Score
0.0raw metric 0.5176
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.5176
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceNo component is trained or fine-tuned. DATA_MANIFEST.md says no training was done and that only the 25 /task/public exam
No component is trained or fine-tuned. DATA_MANIFEST.md says no training was done and that only the 25 /task/public examples were used, for development and tuning. It also declares where every artifact came from: FLUX.1-schnell from Hugging Face at revision 741f7c3c, the PP-OCRv5 detector via validate.py --prepare-models with a sha256, Noto fonts from google/fonts under OFL, and wheels from PyPI. logs/download.sh keeps the download commands. logs/flux_download.log shows the pinned snapshot completing. trajectory#194 shows the snapshot being copied into the output. The trusted wheel audit approves the onnxruntime wheel and gives a PyPI provenance URL. No undeclared corpus appears.
✓non specialist initializationServing loads the general-purpose public FLUX.1-schnell checkpoint frozen, at the pinned revision. In models/FLUX.1-schn
Serving loads the general-purpose public FLUX.1-schnell checkpoint frozen, at the pinned revision. In models/FLUX.1-schnell/transformer/config.json, guidance_embeds is false. The weight inventory holds standard FLUX transformer tensors and no guidance embedder, which matches schnell, not dev. A search found no trace that any FLUX.1-dev comparison weights were shipped: the dev model was downloaded only for experiments in trajectory#215-#221, and the final file list in trajectory#257 shows only schnell. The other components are a general text detector (PP-OCRv5 det, used for erasing only) and a font renderer.
✓no reference image lookupserver.py builds every image procedurally. mockup.py draws a layout wireframe from the request texts and description; FL
server.py builds every image procedurally. mockup.py draws a layout wireframe from the request texts and description; FLUX img2img then runs with a seed derived from a hash of the payload; erase.py cleans up pseudo-text; render.py overlays the exact text with fonts. No stored screenshots, reference images or lookup tables ship: the final file list in trajectory#267 contains only code, fonts, logs, wheels and weights. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and the handler makes no network calls.
✓actual inference pathserve.sh installs the vendored wheels and runs `exec python3 server.py`. server.py loads FluxImg2ImgPipeline from models
serve.sh installs the vendored wheels and runs `exec python3 server.py`. server.py loads FluxImg2ImgPipeline from models/FLUX.1-schnell with local_files_only on cuda, then TextRenderer (render.py) and BgCleaner (erase.py, which uses det_onnx.py with models/ocr). Each request goes through build_mockup, PIPE, the cleaner, the renderer and a PNG encode. The self-check log in trajectory#260-#261 shows /health answering in 21.8 s with real FLUX requests at about 2 s each, and the server log prints the FLUX prompts. trajectory#256 confirms the final sources were diff-verified into the output. The cleaner's model path may be doubled, but on failure it degrades gracefully; that is not a substitution of the declared model.
✓harness credentials used only by the harnessThe only API credential used was the participant's own OpenRouter key from `spin openrouter key create` (trajectory#127,
The only API credential used was the participant's own OpenRouter key from `spin openrouter key create` (trajectory#127, #128, #130), used for local judge scoring on a Spin box (e.g. trajectory#266); this is allowed. No harness API keys, BASE_URL variables or env dumps were accessed. The participant grepped the output for secrets before submitting, with an empty result (trajectory#267). The shipped code has no credentials and makes no network calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 32m into the runtook 7m 33s
startup 21.8 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 21.8 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 41.6 s of server time; at that rate grading's 200 take 416 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 2.1 s, max 4.5 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests