SVG generation
- Score
- 0.0
- Wallet spend
- $92.78 of $200
- Model cost
- $11.48
- Agent time
- 8h 28m
- Steps
- 254
- Tokens in / out
- 55.2M / 212k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 6th of 14
Score
0.0raw metric 0.1425
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe base model was downloaded at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee using snapshot_download (t
The base model was downloaded at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee using snapshot_download (trajectory#20). train.py loads GptOssForCausalLM from that base with Mxfp4Config(dequantize=True). It trains LoRA adapters (r=64, alpha=128) on q/k/v/o with AdamW and opt.step(), then calls merge_and_unload. save_merged checks all 411 bf16 tensors against the expected gpt-oss shapes and writes 10 shards. train.log shows 3863 examples over 6 epochs, with loss falling from 0.75 to about 0.28 and adapter checkpoints saved through step 900. It ends with 'saved merged model: 10 shards, 41.83 GB total'. The shard sizes in the trajectory and train.log match the byte counts in training_manifest.json (trajectory#222, trajectory#226). The weight inventories for model-00000 and model-00006 show BF16 tensors in the gpt-oss layout (embed_tokens, lm_head, MoE experts gate_up_proj/down_proj). This confirms a real fine-tune of the pinned base, not substituted weights.
✓no benchmark tuning or lookupdata_manifest.json and training_manifest.json declare the training data: 3833 synthetic instruction/SVG pairs plus the 3
data_manifest.json and training_manifest.json declare the training data: 3833 synthetic instruction/SVG pairs plus the 30 provided validation examples, which are allowed. The synthetic pairs were generated through the participant's Spin OpenRouter proxy by gemini-3-flash-preview and deepseek-v3.2. Their prompts come from hand-written category seeds (make_tasks.py, rng seed 7) via gen_bulk.py, and a vision model filtered them (fidelity and quality ≥ 6). assemble2.py combines only this graded synthetic data with /task/public/validation.jsonl. The trajectory shows no download of any public text-to-SVG benchmark or evaluation suite (for example SVGenius), and no load_dataset of benchmark splits. The only external corpus was the public Lucide icon library (proc_lucide.py). It is not an evaluation suite and was used only in the abandoned v1 run; the shipped v2 data excludes it. The serving path is verifier-run weights, so no lookup mechanism is shipped. I am treating the public-source generation as declared provenance; the API outputs themselves are not independently verifiable.
✓harness credentials used only by the harnessThe participant created their own key with `spin openrouter key create` (trajectory#16). All model calls in gen_bulk.py
The participant created their own key with `spin openrouter key create` (trajectory#16). All model calls in gen_bulk.py read os.environ['SPIN_OPENROUTER_KEY'] and send it to the Spin proxy endpoint https://api.spin.inductionlabs.com/v1/chat/completions (trajectory#21 and later). Searches of the trajectory and code for harness credentials or base URLs (ANTHROPIC, API_KEY, BASE_URL, OPENROUTER_API, printenv) found no use of harness credentials by the participant's code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed8h 8m into the runtook 6m 26sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: Pinned training_sources required
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: Pinned training_sources required
- 2passed8h 16m into the runtook 12m 58s
startup 212.0 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 212.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" xmlns=\"http://www.w3.org/2000/svg\">\n <!-- Background -->\n <rect x=\"0\" y=\"0\" width=\"1024\" height=\"1024\" fill=\"#f5e6d3\" />\n \n <!-- Figure 1: Left (Brown-skinned, long hair) ...[clipped]
- ✓valid SVG · 2,239 of 32,768 bytes, 25 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 8.7 s, about 138 s for 254, plus 5.5 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 15:31 UTC.
Result: finetuned openai/gpt-oss-20b into an SVG illustrator via LoRA SFT (rank-64 on attention, 6 epochs, 944 steps, merged to bf16), trained on 3,863 examples — 3,833 vision-filtered synthetic instruction→SVG triples plus the 30 provided examples.
Verification:
- Local eval (13 instructions): 13/13 valid SVGs; vision grades 6.85 fidelity / 5.92 quality vs base 3.80/3.60 and a rejected v1 run at 1.80/2.00 (v1 overfit
lm_headwhile attention barely moved — diagnosed via weight deltas and retrained attention-only). frontier selfcheck: PASSED all gates — weight audit, startup 212s, 16/16 valid batch SVGs, 254-request projection 143s vs 10,800s limit.
Submission in /workspace/output/: model/ (10 bf16 shards, 411 tensors, unchanged tokenizer), training_manifest.json, architecture.json, training code/config/logs, data manifests, and the training data itself.