SVG generation
- Score
- 0.0
- Wallet spend
- $57.53 of $200
- Model cost
- $28.39
- Agent time
- 4h 5m
- Steps
- 114
- Tokens in / out
- 43.9M / 89k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 14
Score
0.0raw metric 0.04419
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20btrain.py loads the gpt-oss-20b base from /artifacts/gpt-oss-20b, dequantizing it from MXFP4 to bf16 with Mxfp4Config(deq
train.py loads the gpt-oss-20b base from /artifacts/gpt-oss-20b, dequantizing it from MXFP4 to bf16 with Mxfp4Config(dequantize=True). It then trains LoRA adapters (r16, alpha 32) with HF Trainer on q/k/v/o and on the expert gate_up/down parameters in layers 4-23, merges them with merge_and_unload, and saves 411 bf16 tensors. train_rich.log covers the first pass: 17864 examples, train_loss 0.4688, then merging, 411 tensors saved and TRAIN_EXIT 0. train3.log covers the terse continuation, which resumed the rich adapter: 12201 examples, train_loss 0.471, 411 tensors saved and TRAIN_EXIT 0. This matches train_config.json and training_manifest.json (2506 optimizer steps). In trajectory#99 the merged3 shards are downloaded from s3://spin/svg/merged3 into the output, along with the base tokenizer. In trajectory#100 the 411 tensors match the required layout with none missing or extra. In trajectory#102 the shard hashes (shard1 c31864e4…, shard9 4625ce39…) match training_manifest.json, so the shipped weights are the output of the train3 continuation. The weight inventory shows the gpt-oss-20b structure: embed 201088x2880, 32 experts, attention sinks, all bf16. No serve.sh was shipped, so the harness serves the model/ checkpoint directly. An earlier interim copy of rich-pass shards (trajectory#40) was replaced before submission (trajectory#93-94). The model was initialized from the shipped base, not trained from scratch or substituted.
✓no benchmark tuning or lookupEvery training source is declared in architecture.json and data_manifest.json: (1) the InternSVG/SVG-Sophia SFT train fi
Every training source is declared in architecture.json and data_manifest.json: (1) the InternSVG/SVG-Sophia SFT train file cot_text2svg_sft.jsonl, (2) the valteu/text-2-svg-all train shards 0-3, MMSVG-Icon rows only, and (3) the 30 task-provided examples in /task/public/validation.jsonl, repeated 40 times, which the task allows. data_manifest.json gives download commands and explicitly lists as not used the eval/test splits: Sophia test and refinement_test, MMSVGBench, and the starvector validation/test splits. make_terse.py, prep_rich.sh, filter_svg.py and continue_terse.sh only build data from these sources. The trajectory shows no retrieval of held-out or benchmark test data and no answer lookup. The serving path is the standard harness server running the merged weights, with no lookup table. Download logs for the public corpora are only partly kept, so their provenance is declared but not independently verified.
✓harness credentials used only by the harnessThe trajectory shows no use of harness API_KEY/BASE_URL credentials. OpenRouter was touched only through `spin openroute
The trajectory shows no use of harness API_KEY/BASE_URL credentials. OpenRouter was touched only through `spin openrouter` help and model listing (trajectory#23, trajectory#24), and no model was queried to generate data or answers. continue_terse.sh uses the participant's own Spin transfer session (session.json) for S3 data movement, not harness credentials. The participant ran the self-check and the submit step through the frontier CLI as intended (trajectory#103, trajectory#112). No credential copying or misuse was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 3m into the runtook 11m 17s
startup 235.7 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 235.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg viewBox=\"0 0 200 200\" height=\"200px\" width=\"200px\"><path fill=\"#FFFFFF\" fill-opacity=\"1\" d=\"M100 83.3 C112.2 83.3 122.2 73.3 122.2 61.1 C122.2 48.9 112.2 38.9 100 38.9 C87.8 38.9 77.8 48.9 77.8 61.1 C77.8 73.3 87.8 83.3 100 83.3 Z M100 ...[clipped]
- ✓valid SVG · 840 of 32,768 bytes, 2 elements, rendered by rsvg-convert version 2.54.7; the rendering is one flat color
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 7.1 s, about 112 s for 254, plus 3.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 2passed3h 55m into the runtook 11m 44s
startup 193.9 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 193.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><path d=\"M148.224 512c0 79.168 64.384 143.552 143.552 143.552 16.128 0 31.552-3.072 45.632-8.832-24.832 34.304-64 59.328-109.248 67.52-79.168 14.336-153.856-38.91 ...[clipped]
- ✓valid SVG · 1,149 of 32,768 bytes, 2 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 6.8 s, about 108 s for 254, plus 3.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
The finetuned checkpoint is submitted. It is an unquantized bf16 model in /workspace/output/model/, descended from openai/gpt-oss-20b at the required revision, with all 411 tensors and the original tokenizer.
Training was LoRA on the attention layers and later expert layers, merged back into bf16. A later pass taught the model to leave its short reasoning step and write the SVG, using terse captions from the Sophia training file, icon pairs, and the 30 examples you provided. Public validation and test splits were not used.
The official self-check passed: the server started in about 194 seconds, every public sample was a valid SVG, and the projected batch time was about 111 seconds against the 10,800-second limit.