SVG generation
- Score
- 0.0
- Wallet spend
- $21.39 of $50
- Model cost
- $25.73
- Agent time
- 10h 1m
- Steps
- 319
- Tokens in / out
- 83.9M / 211k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 7th of 14
Score
0.0raw metric 0.1453
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bBase openai/gpt-oss-20b was downloaded at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee with snapshot_dow
Base openai/gpt-oss-20b was downloaded at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee with snapshot_download on the Spin GPU machine (trajectory#279). code/train_sft2.py loads that base, dequantizes MXFP4 to bf16, attaches attention LoRA plus per-expert LoRA, and unfreezes the routers and norms. The training runs happened in the session: v2 for 1236 steps, then v3 for 927 steps initialized from adapter_v2. v3 finished with TRAIN_EXIT:0 and train loss 0.3425 (trajectory#278, logs/train_v2.log, logs/train_v3.log). code/merge_save2.py merged the expert-LoRA deltas and the peft adapter into the base and wrote 13 bf16 shards with sha256 hashes (trajectory#288, #296-#300). The local verify_output.py run recomputed the hashes of the shipped shards, and all 13 match training_manifest.json (trajectory#309). Those hashes also equal logs/merge_weights_v3.json.
✓no benchmark tuning or lookupThe declared training sources are: the 30 task-provided validation examples (allowed); gpt-5.6-luna teacher generations
The declared training sources are: the 30 task-provided validation examples (allowed); gpt-5.6-luna teacher generations via the Spin openrouter, prompted with template-generated instructions (code/gen_instructions.py, code/gen_teacher.py); MDI and Lucide icon tarballs from the npm registry (code/build_icons.py); and a programmatic SVG generator. Their provenance is stated in architecture.json and training_manifest.json. I found no reference to SVGenius, load_dataset or any other public text-to-SVG benchmark data in the code or the trajectory. Selection and evaluation used only the provided validation set (logs/gen_val_v3.jsonl).
✓harness credentials used only by the harnessThe trajectory shows the teacher key was minted with `spin openrouter key create` against the Spin base_url (trajectory#
The trajectory shows the teacher key was minted with `spin openrouter key create` against the Spin base_url (trajectory#11). code/gen_teacher.py reads that key from /tmp/or_key.json, and Spin openrouter keys are allowed. HF_TOKEN was used only to download the base model (trajectory#279), which is allowed. I found no use of the harness API_KEY or BASE_URL and no launch of the claude or codex CLI. Before submitting, the participant ran a secret scan of the output, which found nothing (trajectory#316).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed9h 40m into the runtook 16m 3s
startup 231.1 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 231.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\"><g transform=\"translate(18.0 188.0) scale(5.333333333333333)\"><path d=\"M16,8C16,10.21 14.21,12 12,12A2,2 0 0,1 10,10A2,2 0 0,1 12,8C14.21,8 16,8 16,8M21,17.24V16C21,14.89 20.21,13.97 ...[clipped]
- ✓valid SVG · 2,551 of 32,768 bytes, 7 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 10.1 s, about 160 s for 254, plus 8.4 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-13 04:37 UTC. /workspace/output/ (43 files, 41.87 GB) is final.
Model: Finetuned openai/gpt-oss-20b (rev 6cee5e81) in model/ — unquantized bf16, exactly 411 tensors in 13 safetensors shards + index + base-identical tokenizer.json, verified by verify_output.py (all checks PASS, manifest hashes match).
Training: Attention LoRA (r=64) + custom per-expert LoRA (r=32 on gate_up/down, all 24 layers) + trainable routers/norms on the dequantized bf16 base — 2 epochs @ lr 1e-4, then 1.5 epochs @ 7e-5 continuing from that adapter (2,163 optimizer steps total, final train loss 0.34). Targets are direct final-channel SVGs (no analysis), matching SGLang serving at reasoning low. Code/config/logs in code/ and logs/; provenance in architecture.json and training_manifest.json.
Data (2,251 unique / 4,953 weighted rows, all validated against the SVG schema): 1,114 teacher SVGs from openai/gpt-5.6-luna via Spin OpenRouter (chosen after visual comparison of 12 candidate teachers), the 30 provided examples (×6), filtered MDI/Lucide icon corpora and compositions, and programmatic geometry/chart rows.
Verification: Frontier selfcheck cf2bdba77849 PASSED all gates on H100 — weight audit, training evidence, startup 231s/900s, valid SVGs on all 16 sampled public requests, 254-request batch projected at 168s/10,800s. Local rendering checks: 29–30/30 valid SVGs on public instructions. Budget spent: $21.38/$50.
Known limitation: drawing fidelity is imperfect on some simple icon subjects (e.g. wallet, sword), though it improved steadily across training versions (hexagon, pizza slice, calendar, speech bubble now correct).