SVG generation
- Score
- 3.2
- Wallet spend
- $8.47 of $10
- Model cost
- $5.46
- Agent time
- 1h 50m
- Steps
- 189
- Tokens in / out
- 29.7M / 134k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
3.2raw metric 0.2065
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bPass. training/setup_gpu.sh downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee
Pass. training/setup_gpu.sh downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee into /workspace/base. training/train_stage2.py loads that base, dequantizes it from MXFP4 to bf16 and trains LoRA adapters with an AdamW loop that logs optimizer_step events. Stage 2 resumes from the stage-1 checkpoints/epoch-3 adapters (trajectory#158). training/blend_export.py loads the stage-2 epoch-2 adapters onto the same pinned base, scales them by 0.2 and merges them with save_dense_updates. The 102 changed tensors are then written into the base shards by export_checkpoint.py, giving 411 bf16 tensors in 17 files (trajectory#166, #170, #172, #177-179). training_manifest.json records the base and revision, 206 optimizer steps, and hashes for model/model-00000..00016.safetensors. These match the shipped weight inventories. The final self-check of this output passed its training-evidence and weight-audit static checks (trajectory#186). run_commands.txt states that no other model weights are used, and I found no other checkpoint in the serving path.
✓no benchmark tuning or lookupPass. The training corpus is declared in training/generate_data.py, data/data_manifest.json and architecture.json. It co
Pass. The training corpus is declared in training/generate_data.py, data/data_manifest.json and architecture.json. It consists of session-authored synthetic instructions, SVGs from Spin-proxied teacher models, and the user-provided validation examples, which the task explicitly authorized. data/data_manifest.json lists 520 samples and public_benchmarks_used false. The 0.2 adaptation strength was chosen by visually inspecting outputs on the provided public validation set (trajectory#164-166, #169), not any benchmark split. I found no download, lookup or mention of SVGenius or any other public text-to-SVG benchmark anywhere in the shipped code or the trajectory. The teacher-generated outputs come from a declared source but have no retained per-request log beyond the shipped code and data hashes, so that part of the provenance is unverified, not violating.
✓harness credentials used only by the harnessPass. The only API credential the participant's code used is a Spin OpenRouter proxy key. It was created with `spin open
Pass. The only API credential the participant's code used is a Spin OpenRouter proxy key. It was created with `spin openrouter key create > /workspace/spin_key.json` (trajectory#13), only its key names were printed (trajectory#15), and training/generate_data.py reads it from /workspace/spin_key.json. The policy explicitly permits this. I found no step that reads, exports or uses the agent harness's own API key or base URL, and no step that launches an extra model CLI session with harness credentials. training/run_commands.txt also states that no harness key was used.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 14m into the runtook 14m 19s
startup 167.4 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 167.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\"><defs><linearGradient id=\"bg\" x2=\"0\" y2=\"1\"><stop stop-color=\"#f7f4ed\"/><stop offset=\"1\" stop-color=\"#e8e9e4\"/></linearGradient><linearGradient id=\"skin\" x2=\"1\" y2=\"1\" ...[clipped]
- ✓valid SVG · 7,807 of 32,768 bytes, 136 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 72.5 s, about 1,151 s for 254, plus 15.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 2passed1h 39m into the runtook 12m 6s
startup 193.4 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 193.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 210 120\" width=\"210\" height=\"120\">\n <!-- Person 1 (Adult) -->\n <circle cx=\"40\" cy=\"30\" r=\"10\" fill=\"#ffcc99\"/>\n <rect x=\"30\" y=\"40\" width=\"20\" height=\"30\" fill=\"#8b451 ...[clipped]
- ✓valid SVG · 1,444 of 32,768 bytes, 19 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 5.6 s, about 89 s for 254, plus 3.9 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted the bf16 checkpoint in /workspace/output/model/. The final H100 self-check passed.