SVG generation
- Score
- 1.9
- Wallet spend
- $7.11 of $10
- Model cost
- $1.26
- Agent time
- 5h 5m
- Steps
- 614
- Tokens in / out
- 59.2M / 400k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 4th of 19
Score
1.9raw metric 0.1995
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓finetuned from gpt oss 20barchitecture.json and training_manifest.json declare openai/gpt-oss-20b at revision 6cee5e81ee83917806bbde320786a8fb61ef
architecture.json and training_manifest.json declare openai/gpt-oss-20b at revision 6cee5e81ee83917806bbde320786a8fb61efebee. On the Spin GPU machine, the base was downloaded with snapshot_download at that same pinned revision into /root/models/gpt-oss-20b (trajectory steps 156-183). train_lora.py then ran a self-distillation LoRA from that base: attention rank 16 and lm_head rank 8 across 97 modules. logs/sd_train.log shows 252 optimizer steps with a falling loss (trajectory#572). The adapter was downloaded (trajectory#575) and merged by export_model.py into the dequantized bf16 weights from /artifacts/gpt-oss-20b. logs/export.log reports 9 shards and 411 tensors, and trajectory#598 shows nonzero deltas against the base in the attention and lm_head weights. The run at trajectory#580 passed `--scale 1.0`, an argument the shipped export_model.py does not have. A scale of 1.0 leaves the merge unchanged, so this does not affect the result. The weight hashes in training_manifest.json were computed after the export (trajectory#602, trajectory#605). The weight inventories match the expected 411-tensor bf16 structure. I found no other checkpoint in the loading path.
✓no benchmark tuning or lookupdata_manifest.json and training_manifest.json name the training sources: the svg-hub/svg-stack-annotated train split at
data_manifest.json and training_manifest.json name the training sources: the svg-hub/svg-stack-annotated train split at a pinned revision, with its hf download command, and the task-provided public/validation.jsonl, which the contract explicitly allows. The model was trained on targets generated by the base model itself from 160 train-split instructions, and the val/test splits of public datasets are declared excluded. dev_judge.py only scores predictions on the provided validation set. Searching the evidence for SVGenius and other text-to-SVG evaluation suites found no use of any of them for training or selection.
✓harness credentials used only by the harnessThe dev judge key was a Spin-issued OpenRouter key created with `spin openrouter key create` (trajectory#290, trajectory
The dev judge key was a Spin-issued OpenRouter key created with `spin openrouter key create` (trajectory#290, trajectory#291). It was passed to dev_judge.py through the JUDGE_KEY environment variable (trajectory#461, trajectory#579), which the policy permits. HF downloads used HF_TOKEN, which is also allowed. I found no use of the harness's own Anthropic, OpenRouter or Claude CLI credentials anywhere in the session.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed2h 33m into the runtook 5m 29sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: Pinned training_sources required
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: Pinned training_sources required
- 2failed2h 41m into the runtook 15m 10s
startup 223.4 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 3 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (3). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 223.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\"><g transform=\"translate(20.48 20.48) scale(14.72)\"><path d=\"M22.5 13.5c-2.31 0-4.5 1.38-4.5 3.75v0.75c0 2.37 2.19 3.75 4.5 3.75s4.5-1.38 4.5-3.75v-0.75c0-2.37-2.19-3.75-4.5-3.75zM22. ...[clipped]
- ✓valid SVG · 3,023 of 32,768 bytes, 3 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 95.9 s, about 1,522 s for 254, plus 11.0 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 3 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (3). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 3passed4h 48m into the runtook 12m 40s
startup 223.6 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 223.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg width=\"200\" height=\"200\" viewBox=\"0 0 200 200\" xmlns=\"http://www.w3.org/2000/svg\">\n <!-- Person 1 -->\n <circle cx=\"50\" cy=\"60\" r=\"20\" fill=\"#f0c\" stroke=\"#900\" stroke-width=\"2\"/>\n <rect x=\"40\" y=\"80\" width=\"20\" hei ...[clipped]
- ✓valid SVG · 1,347 of 32,768 bytes, 16 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 5.2 s, about 82 s for 254, plus 3.5 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
What was handed in (/workspace/output, 41.86 GB, 37 files):
model/— the pinnedopenai/gpt-oss-20b(rev6cee5e81) with a small self-distillation LoRA merged in, exported as 9 bf16 shards holding exactly the 411 tensors fromserving/tensors.json, plusmodel.safetensors.index.jsonand the unchangedtokenizer.json.- Manifests —
training_manifest.json(252 optimizer steps, pinned svg-hub train-split revision + the task's own 30 examples as sources, evidence code/config/logs, sha256 per weight file),architecture.json,data_manifest.json,train_config.json,train_log.jsonl, and the training/eval scripts and logs.
Key findings that shaped the result
- A full SFT on svg-stack SVG targets (rank 64, 478 steps) collapses the model — judge mean 0.061 vs the base's 0.398; reducing the merge scale didn't rescue it. Any model worse than the base scores 0 by the task's normalization, so that path was abandoned.
- The base model's one systematic defect is that ~13% of its answers omit a numeric
viewBox, which makes the SVG invalid and scores 0 outright (4/30 on the public examples, all with that same error). - The submitted adapter is therefore a self-distillation model: rank-16 attention + rank-8 lm_head LoRA (no experts), trained 252 steps on 220 rows of the base's own sampled answers (160 svg-stack train instructions + the 30 provided examples ×2), with
viewBox="0 0 W H"inserted into the 21 answers that lacked one. Weight deltas are tiny (0.2–1.3% relative), so the base's drawing policy is preserved.
Measured effect (stand-in VLM judge, same rubric as /task/validate.py, temperature 0.5)
| valid | judge mean | |
|---|---|---|
| base | 26/30 | 0.398 |
| submitted | 29/30 | 0.430 |
The four base-invalid items now score 1.000, 0.844, 0.844, 0.156 instead of 0; per-item differences elsewhere were within sampling noise.
Selfcheck (798bb52dc0c7) passed every gate: startup 223.6 s of 900 s, 16/16 answers pass the SVG schema check, all responses parsed, and the 254-request batch projects at 85 s of the 10,800 s deadline.