Screenshot generation
- Score
- 59.7
- Wallet spend
- $30.73 of $50
- Model cost
- $38.57
- Agent time
- 8h 53m
- Steps
- 518
- Tokens in / out
- 52.8M / 388k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 2nd of 14
Score
59.7raw metric 0.7861
The raw metric (mean of quality (OCR text, OCR layout, judged faithfulness) and judged realism) is rescaled so that a baseline that draws each text line into its box on a white page (0.537) scores 0 and the real screenshots (0.954) scores 100, clamped to that range.
Verifier details
- metric
- mean_accuracy_realism
- samples
- 200
- available_samples
- 500
- judge_model
- gpt-6-luna
- ungated_score
- 0.7861
- invalid_images
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceDATA_MANIFEST.md declares the training images: public OS-Copilot/OS-Atlas-data zips (mind2web_images, guiact_images, UIB
DATA_MANIFEST.md declares the training images: public OS-Copilot/OS-Atlas-data zips (mind2web_images, guiact_images, UIBert, desktop linux/macos/windows) and xlangai/AgentNet. The download scripts and logs were kept. Captions were made with a VLM through the participant's own Spin OpenRouter proxy. The shipped public validation examples were used only for dev evaluation and choosing settings, which is allowed. The theme classifier was trained on the participant's own captions, and calibration used self-generated synthetic text. The trajectory shows no access to the hidden or evaluation dataset repo. The Mind2Web and UIBert images come from OS-Atlas training bundles, and nothing shows use of a test split.
✓non specialist initializationThe shipped transformer is the general-purpose FLUX.2-klein-base-4B at revision a3b4f48…. It was downloaded with snapsho
The shipped transformer is the general-purpose FLUX.2-klein-base-4B at revision a3b4f48…. It was downloaded with snapshot_download (trajectory#431), and trajectory#474 shows it copied into output/model. Run b1 trained a new PEFT LoRA starting from that base snapshot (trajectory#434: --model $S on the base snapshot, --out ~/lora_b1). That LoRA is what ships as lora/lora_final.safetensors (trajectory#493, #494). No screenshot-specialist checkpoint was used for initialization.
✓no reference image lookupserver.py sends requests to pipeline.Generator.generate. That function builds a text-only skeleton (gen/skeleton.py), ru
server.py sends requests to pipeline.Generator.generate. That function builds a text-only skeleton (gen/skeleton.py), runs the FLUX image-edit pipeline, inpaints stray text and re-renders the requested text with bundled fonts. The fallback is just the skeleton render. There is no network access, no stored screenshot corpus and no lookup by description or nearest neighbour.
✓actual inference pathserve.sh sets HF_HUB_OFFLINE=1 and runs server.py. server.py calls gen/pipeline.Generator, which loads Klein(model/, lor
serve.sh sets HF_HUB_OFFLINE=1 and runs server.py. server.py calls gen/pipeline.Generator, which loads Klein(model/, lora/lora_final.safetensors) through Flux2KleinPipeline.from_pretrained(local_files_only=True) and merges the LoRA into the linear weights (gen/klein_infer.py). It also loads the bundled PP-OCRv5 ONNX detector. These shipped weights are what produce the responses.
✓harness credentials used only by the harnessThe only API credentials in the trajectory came from `spin openrouter key create` (trajectory#16, #17), the participant'
The only API credentials in the trajectory came from `spin openrouter key create` (trajectory#16, #17), the participant's own Spin proxy allowance. They were used for captioning and dev scoring. HF_TOKEN was used only for downloads, and nothing was uploaded. No harness or judge credentials were used, and no extra model CLI sessions were started.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed2h 34m into the runtook 2m 15sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed8h 46m into the runtook 8m 48s
startup 41.6 s of 300 s; 3/3 gates passed; The speed gates were measured on a sample: 20 sequential public requests against their 720 s share of grading's 7,200 s for 200, the budget projected to all 200 and p95 and the failure limit taken over the 20.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 41.6 s
- ✓failed requests · none of the 20 requests failed, timed out or answered anything but one 1280x720 8-bit RGB PNG
- ✓generation budget (projected) · projected from 20 of 200 requests at concurrency 1: they took 166.7 s of server time; at that rate grading's 200 take 1,667 s of the 7,200 s it allows
- ✓p95 latency · 20 sequential public requests, the first as soon as /health answered: mean 8.3 s, max 9.1 s; p95 of 20 allows 1 above 30 s as grading's p95 of 200 allows 10; clock: trusted HTTP client inside the serving sandbox; grading transport (provider transfer/exec) excluded
- ✓valid images · all 20 answers were valid 1280x720 RGB PNGs the scorer decodes
- ✓healthy after requests
The agent's last message before its session ended.
I submitted /workspace/output, and the self-check passed on that exact package: 119 files, 16.85 GB. It reports no score, and the architecture review did not run, so I can't rule out a zero from that review.
My own dev scoring on the 25 public validation examples gave a 3-seed mean of about .80 for the shipped LoRA, against .714 for the earlier distilled model. Three-seed scores per seed were .818, .779 and .800.
What ships
- Base model: FLUX.2-klein-base-4B, with a rank-64 LoRA I trained myself.
- Generation: the model turns a text-only wireframe of the requested boxes plus the description into a screenshot, using 20 steps and guidance 5.
- Post-processing: stray or garbled text is detected, inpainted away, and the requested texts are re-rendered exactly in their boxes.
Self-check gates
- Startup:
/healthanswered after 41.6 s (limit 300 s). - Failures: 0 of 20 requests failed.
- Latency: p95 was 9.1 s (limit 30 s), mean 8.3 s.
- Total time: projected 1667 s for 200 requests (limit 7200 s).
- Image format: all 20 images were valid 1280x720 RGB PNGs.
Two things to know
- The full console log of the shipped training run was lost when the rented H100 expired before I copied it back.
train/logs/phase3_base/README_excerpt.txtholds the progress lines I had seen while polling, plus the exact command;DATA_MANIFEST.mdstates this. - I wrote the 3-seed result and the final data manifest around the check. The manifest and
train/COMMANDS.mdhad already been updated before the check started. The A100 was stopped and the H100 had already expired.