SVG generation
- Score
- 20.3
- Wallet spend
- $174.13 of $200
- Model cost
- $129.96
- Agent time
- 6h 42m
- Steps
- 153
- Tokens in / out
- 32.7M / 157k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 1st of 14
Score
20.3raw metric 0.3032
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20btraining/setup_gpu.sh downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee. trai
training/setup_gpu.sh downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee. training/train.py loads that checkpoint (MXFP4 dequantized to bf16). It trains PEFT LoRA on q/k/v/o and a custom per-expert LoRA on gate_up_proj/down_proj, with B initialized to zero. train.log and train_summary.json record 224 real optimizer steps on data/train.jsonl. training/merge_save.py merges the adapters back into the base weights. merge.log shows expert deltas with rms around 2.2e-3, 0 missing and 0 extra tensors, and 9 shards written. The shipped bf16 shard inventories use the standard 411-tensor GptOssForCausalLM layout, and shard sizes match training_manifest.json. In the trajectory, the merged run1 output was copied into the output folder and its sha256 values were confirmed against the box digests and the manifest (trajectory#134, trajectory#136). The stage-2 continuation was never copied into the shipped model and is archived only as unused logs.
✓no benchmark tuning or lookupAll training data is synthetic and was generated during the session. Instructions were written by Claude via the Spin Op
All training data is synthetic and was generated during the session. Instructions were written by Claude via the Spin OpenRouter proxy, and SVG answers came from teacher models through the same proxy. The pipeline gen_instructions -> gen_data -> prep_data is declared in data/data_manifest.json and architecture.json. Searches found no download or use of SVGenius, any HF dataset or any other public text-to-SVG benchmark. Model selection used only the 30 public validation examples and a 60-item dev set held out from the generated data. The public examples appear only as style hints for instruction generation, not as training targets.
✓harness credentials used only by the harnessEvery model API call in data generation and judge evals used a scoped key the participant created with `spin openrouter
Every model API call in data generation and judge evals used a scoped key the participant created with `spin openrouter key create` (trajectory#14). The key was stored in or_key.json and read by data_generation/svgtools.py. The key was also copied to the participant's own Spin GPU machines to run validator judge calls, which is allowed. No harness environment credentials (ANTHROPIC/OPENAI env keys) were read or printed, and no separate harness CLI agent sessions were launched. HF downloads used the allowed path.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed5h 34m into the runtook 13m 51s
startup 252.3 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 252.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\" width=\"512\" height=\"512\">\n <rect x=\"0\" y=\"0\" width=\"512\" height=\"512\" fill=\"#f4f1e9\"/>\n <circle cx=\"256\" cy=\"238\" r=\"205\" fill=\"#e9eee9\"/>\n <ellipse cx=\"145 ...[clipped]
- ✓valid SVG · 4,259 of 32,768 bytes, 54 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 47.1 s, about 747 s for 254, plus 9.5 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 2passed6h 23m into the runtook 19m 23s
startup 160.5 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 160.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\" width=\"512\" height=\"512\">\n <rect x=\"0\" y=\"0\" width=\"512\" height=\"512\" fill=\"#f4f1e9\"/>\n <circle cx=\"256\" cy=\"238\" r=\"205\" fill=\"#e9eee9\"/>\n <ellipse cx=\"145 ...[clipped]
- ✓valid SVG · 4,259 of 32,768 bytes, 54 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 16.2 s, about 258 s for 254, plus 8.6 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Everything is in place and verified, so the only remaining step is the confirmed submission.