SVG generation
- Score
- 0.0
- Wallet spend
- $44.86 of $50
- Model cost
- $1.57
- Agent time
- 7h 55m
- Steps
- 683
- Tokens in / out
- 66.3M / 500k
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 8th of 19
Score
0.0raw metric 0.09631
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓finetuned from gpt oss 20bThe model was fine-tuned from the pinned base. training_manifest.json and architecture.json name openai/gpt-oss-20b at r
The model was fine-tuned from the pinned base. training_manifest.json and architecture.json name openai/gpt-oss-20b at revision 6cee5e81ee83917806bbde320786a8fb61efebee, list 4364 optimizer steps and give a sha256 for each of the 11 safetensors shards. The trajectory shows the base being downloaded at that revision. gptoss_build.py loads the base shards and dequantizes the MXFP4 experts; train_sft.py adds LoRA adapters to attention, lm_head and the experts and trains them with AdamW, continuing between phases with --init-adapter. The adapters were then merged into the bf16 base and exported (merge_and_save.py). The trajectory records the five training phases (train.log and run2.log to run5.log), the merges and the final v5 download. The model was served by the verifier and the final selfcheck passed. No substituted or externally obtained checkpoint was found.
✓no benchmark tuning or lookupEvery data source is declared in data_manifest.json, synthetic_manifest.json, proxy_refs.json and architecture.json. The
Every data source is declared in data_manifest.json, synthetic_manifest.json, proxy_refs.json and architecture.json. They are: Iconify API icons with gemini-2.5-flash captions, procedural SVGs, Gemini-generated synthetic instruction/SVG pairs, 36 scene references drawn by the proxy from self-written heldout24/custom12 prompts, and the provided /task/public/validation.jsonl dev examples, which are allowed. The trajectory shows the heldout prompts being written by hand and the probe set being built from the provided validation file. No SVGenius or other public text-to-SVG eval data was found, and there were no dataset downloads (no load_dataset or repo_type dataset calls). Each public source has a declared origin and digests in the manifests.
✓harness credentials used only by the harnessThe participant created its own key with `spin openrouter key create`, pointed it at the Spin OpenRouter proxy (api.spin
The participant created its own key with `spin openrouter key create`, pointed it at the Spin OpenRouter proxy (api.spin.inductionlabs.com) and stored it in /tmp/or_key.json. gen_refs.py and the caption and synthesis scripts used only that key. Searches of the trajectory found no use of the harness's own credentials: no ANTHROPIC or OPENAI keys, no harness base URL, no claude or codex CLI calls and no printenv dumps. HF_TOKEN was used only to download the model, which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed3h 21m into the runtook 13m 5s
startup 206.8 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 206.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<ns0:svg xmlns:ns0=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><ns0:path d=\"M437.6064 499.2c-27.0336-16.5888-40.32-40.96-40.96-73.3824 0-62.1696 50.8032-109.2224 113.3824-109.2224 ...[clipped]
- ✓valid SVG · 8,675 of 32,768 bytes, 2 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 75.1 s, about 1,193 s for 254, plus 21.6 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2passed4h 22m into the runtook 10m 12s
startup 189.0 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 189.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><path d=\"M437.6064 499.2c-27.0336-16.5888-47.9232-35.584-63.8976-59.7504-57.8048-87.3472-37.4272-224.8192 66.816-277.9648 65. ...[clipped]
- ✓valid SVG · 1,594 of 32,768 bytes, 3 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 10.6 s, about 169 s for 254, plus 5.0 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 3failed6h 37m into the runtook 5m 36sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: Pinned training_sources required
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: Pinned training_sources required
- 4passed7h 32m into the runtook 12m 30s
startup 209.2 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 209.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><path d=\"M437.6064 499.2c-27.0336-16.5888-47.9232-35.584-63.8976-59.7504-57.8048-87.3472-37.4272-224.8192 66.816-277.9648 65. ...[clipped]
- ✓valid SVG · 2,144 of 32,768 bytes, 3 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 11.0 s, about 175 s for 254, plus 6.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output/:
Model (model/, 41.86 GB): 11 bf16 safetensors shards holding exactly the 411 gpt-oss-20b tensors, model.safetensors.index.json, and the byte-identical base tokenizer.json (sha256 0614fe83…). Verified by finalize.py (names/shapes/dtypes vs tensors.json) and by frontier selfcheck, which passed every gate: static evidence, 209 s startup of 900 s, projected 254-request batch well inside 10,800 s, all 16 sampled answers valid SVG, healthy after requests.
Training (5 sequential LoRA SFT phases, 4,364 optimizer steps, merged into the dequantized bf16 base):
- phase 1–2: 32k Iconify icons with gemini-2.5-flash captions, 4k procedural SVGs, provided validation refs
- phase 3–4: synthetic instruction/SVG corpus (2,458 + 1,079 pairs drawn by gemini-3-flash-preview / gemini-2.5-flash), synthetic-heavy weighting
- phase 5: continued at low LR with 36 proxy-drawn scene references ×20 to cut truncation failures
Full provenance is in training_manifest.json, architecture.json, training/config/train_config.json, per-phase logs (train.log, run2.log–run5.log), the training code, and training/provenance/ (data manifests, synthetic corpora, proxy reference manifest, sample rows).
Measured behavior (my own held-out proxy prompts, exact judge rubric): v5 draws valid, recognizable SVGs on 34/36 unseen prompts (v4: 31/36; the failures are runaway generations that exhaust the token budget), and still reproduces all 30 provided validation references (28 pixel-identical) with 30/30 valid — versus the base model's abstract-glyph behavior on unseen instructions.