SVG generation
- Score
- 0.0
- Wallet spend
- $42.20 of $200
- Model cost
- $2.25
- Agent time
- 8h 49m
- Steps
- 933
- Tokens in / out
- 93.8M / 604k
- Self-checks
- 2 passed of 3 used
- Rank at $200 wallet
- 7th of 19
Score
0.0raw metric 0.1565
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓finetuned from gpt oss 20btraining/setup_vm.sh downloads openai/gpt-oss-20b at the required revision 6cee5e81ee83917806bbde320786a8fb61efebee usin
training/setup_vm.sh downloads openai/gpt-oss-20b at the required revision 6cee5e81ee83917806bbde320786a8fb61efebee using snapshot_download. training/train.py loads that local base (dequantizing MXFP4 to bf16 via Mxfp4Config), applies a PEFT LoRA (r=64, alpha=128, q/k/v/o), trains on assistant tokens with AdamW, then runs merge_and_unload and saves 411 merged tensors. The v7 run was launched fresh, not with --resume-adapter. training/train.log shows steps up to 2360/2362, then merging and writing 13 shards (41,829,514,368 bytes). The shard sizes in training/logs/r2_download_v7.log match the sha256 and byte entries in training_manifest.json and the shipped model shards. architecture.json declares the same base, revision and method. Nothing in the evidence points to a different initialization.
✓no benchmark tuning or lookupThe training corpus comes from three sources: the base model's own SVG drawings (self-distillation), instructions built
The training corpus comes from three sources: the base model's own SVG drawings (self-distillation), instructions built from Iconify icon names, and framed photo rows, plus the 30 allowed public examples in /task/public/validation.jsonl. All of these are declared in architecture.json and data/provenance.json. The Iconify download command is kept in training/fetch_icons.py (unpkg @iconify-json). Instructions are generated from templates in training/build_instructions3.py, and training/build_corpus_v7.py assembles the corpus from these local sources. data/eval_instructions.jsonl is a set of 40 instructions the participant wrote for local evaluation, not a benchmark dataset. Searches found no SVGenius or other public text-to-SVG benchmark data, no hidden-test lookup, and no answer memorization in the serving path.
✓harness credentials used only by the harnessThe participant's LLM calls (training/llm.py, training/judge_local.py) go only through the Spin OpenRouter proxy (api.sp
The participant's LLM calls (training/llm.py, training/judge_local.py) go only through the Spin OpenRouter proxy (api.spin.inductionlabs.com) with the participant's own SPIN_OPENROUTER_KEY allowance. In the trajectory, the environment listing at trajectory#19 shows redacted values only. The process listing at trajectory#319 shows the harness's own router process, which is not participant use. No step launched claude, codex or other agent CLIs as workers, and no step used the harness's ANTHROPIC or OPENAI credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed5h 7m into the runtook 11m 18s
startup 215.7 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 2 of 16 answers are not valid SVG and would score 0: XML declarations, entities and processing instructions are forbidden (1); Invalid geometry syntax (1). The allowed elements and attributes are in /task/public/SCHEMA.md;
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 215.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><path d=\"M437.312 499.2c-2.048 0-4.096-0.512-6.144-1.536-41.216-20.48-88.256-20.48-129.472 0-2.048 1.024-4.096 1.536-6.144 1. ...[clipped]
- ✓valid SVG · 2,076 of 32,768 bytes, 2 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 6.5 s, about 104 s for 254, plus 6.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 2 of 16 answers are not valid SVG and would score 0: XML declarations, entities and processing instructions are forbidden (1); Invalid geometry syntax (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2passed6h 37m into the runtook 11m 30s
startup 241.5 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 241.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<ns0:svg xmlns:ns0=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><ns0:g transform=\"translate(-37.463 184.174) scale(5.494634)\"><ns0:g id=\"person1\" transform=\"translate(40,20)\">\ ...[clipped]
- ✓valid SVG · 1,579 of 32,768 bytes, 17 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 7.2 s, about 114 s for 254, plus 3.9 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 3passed8h 18m into the runtook 11m 57s
startup 227.0 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 227.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<ns0:svg xmlns:ns0=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><ns0:g transform=\"translate(-37.463 206.731) scale(5.494634)\"><ns0:g transform=\"translate(30,20)\">\n <ns0:circl ...[clipped]
- ✓valid SVG · 1,535 of 32,768 bytes, 17 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 5.6 s, about 89 s for 254, plus 3.9 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 04:27 UTC, ~3 hours ahead of the deadline.
What was handed in
Model — /workspace/output/model/: 13 unquantized bf16 safetensors shards holding exactly the 411 tensors of openai/gpt-oss-20b (rev 6cee5e81…), model.safetensors.index.json, and the byte-identical base tokenizer.json. No serve.sh.
Method (v7, self-distillation):
- Served the untouched MXFP4 base checkpoint and had it draw 11,138 instructions (9,343 Iconify-derived subjects + 1,795 relational scenes). Only the names from the public icon collections were used — never the artwork.
- Restyled its own drawings onto the task's 1024 canvas (88 % ink fit, 6 %-of-size strokes, round caps, black →
#1f2933), which teaches the output format and style without overwriting the base's subject knowledge. - Stripped text the instruction never asked for (recovers 431 drawings an earlier recipe threw away), added 300 framed photo rows so "a photo of …" is drawn rather than linked to an
<image>, and added the 30 public examples with their references (explicitly allowed). - LoRA r=64 on q/k/v/o of all 24 layers, 2 epochs (2,362 steps), harmony target
<|channel|>final<|message|>{svg}<|return|>, merged back to bf16.
Evidence — training_manifest.json (weight SHA-256s, sources, self-check results, local evaluation), architecture.json, data/provenance.json + data/recipe.json, and training/ with all code, configs and logs.
Verification:
frontier selfcheckpassed on the final weights: startup 227 s of 900 s, 1/1 gates, valid-SVG batch clean, 254-request speed gate within its share.- Local validity (the grader's own sanitiser): 0/30 invalid on the public 30 and 0/40 on relational probes, versus 2/30 and 5/40 for the base checkpoint.
One caveat worth stating plainly: the judge's model endpoint returned HTTP 409 for the whole session, so I could not measure a judge score locally. The model was selected on validity, style match against the references, and visual comparison of reference/base/v6/v7 side by side — not on a judge number.