SVG generation
- Score
- 0.0
- Wallet spend
- $8.65 of $10
- Model cost
- $15.47
- Agent time
- 3h 18m
- Steps
- 243
- Tokens in / out
- 22.3M / 135k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 4th of 14
Score
0.0raw metric 0.08218
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe shipped weights are the pinned base, finetuned in the session. dequantize.py loads the provided /artifacts/gpt-oss-2
The shipped weights are the pinned base, finetuned in the session. dequantize.py loads the provided /artifacts/gpt-oss-20b base and converts its MXFP4 experts to the 411-tensor bf16 layout. train_sft.py loads that base and trains only LoRA parameters on q/k/v/o. train.log shows 1250 steps, 96 wrapped modules and falling loss. merge_lora.py adds the B@A*alpha/r deltas into the base shards. In the session: training ran on the A100 (trajectory#203), the LoRA files were downloaded (trajectory#222) and merged (trajectory#223). Verification showed 96 tensors changed and 315 identical to the base (trajectory#224), which matches the trusted weight audit. After the manifest was first written, only config.json changed: quantization_config was stripped at trajectory#228. The manifest was regenerated at trajectory#232. At trajectory#236 every packaged weight and config hash matched training_manifest.json. architecture.json declares openai/gpt-oss-20b at revision 6cee5e81.
✓no benchmark tuning or lookupThe training corpus is declared in training/data_manifest.json, training_manifest.json and architecture.json. It has fou
The training corpus is declared in training/data_manifest.json, training_manifest.json and architecture.json. It has four parts: instructions written by gpt-oss-120b from a hand-written topic bank (gen_instructions.py), completions distilled from gemini-3.1-flash-lite, the 30 shipped public validation examples (which are allowed), and filtering by ling-3.0-flash-vl scores. The trajectory shows no SVGenius or other public text-to-SVG eval suite being downloaded or used: no load_dataset, HF dataset, git clone or wget of benchmark data. Model selection and tests used only the public inputs. These generation sources are declared but were not independently verified beyond the shipped scripts and logs.
✓harness credentials used only by the harnessTeacher and judge API calls went through the Spin OpenRouter proxy, which is allowed. They used a wallet key the partici
Teacher and judge API calls went through the Spin OpenRouter proxy, which is allowed. They used a wallet key the participant created with `spin openrouter key create` (trajectory#18) and stored in /workspace/.spinkey (trajectory#19); api.py reads SPIN_KEY or that file. The trajectory shows no harness API keys or base URL being read, copied or used, and no extra harness CLI launches. GPU work used spin compute.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed1h 44m into the runtook 10mfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json at the top of the submission
- 2passed3h 6m into the runtook 11m 40s
startup 168.2 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 168.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <circle cx=\"256\" cy=\"256\" r=\"200\" fill=\"#FFD700\" stroke=\"#DAA520\" stroke-width=\"8\"/>\n <g transform=\"translate(160, 180)\">\n <circle cx=\"40\" cy=\"60\" r=\"40\" f ...[clipped]
- ✓valid SVG · 834 of 32,768 bytes, 14 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 9.3 s, about 147 s for 254, plus 3.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of what was built:
Pipeline
- Dequantization: gpt-oss-20b's MXFP4 experts → bf16 (411-tensor unquantized layout), hash-verified between CPU and GPU runs.
- Data: 3,363 instructions (simple/medium/complex) generated in-session by gpt-oss-120b; SVG+plan completions from gemini-3.1-flash-lite (2,994 collected after budget-stopping at 2,634 + 360 complex); quality-filtered per stratum with ling-3.0-flash-vl rendered-image scores (bottom 30% dropped, complex ×2 oversampled) → 2,367 rows including the 30 provided reference examples.
- Training: LoRA r=96 α=192 on all attention projections (q/k/v/o × 24 layers), harmony format with the exact serving system prompt (reasoning: low, randomized dates), loss on completion tokens only — 1,250 steps, ~36 min on the A100.
- Merge: LoRA deltas folded additively into the dequantized base locally on CPU, writing the 6 bf16 shards + index into
output/model/(96 tensors changed, max |Δ| 0.125; 411 tensors match tensors.json; tokenizer.json byte-identical to artifacts).
Verification
- End-to-end generation tests both on GPU (base+LoRA) and locally on the merged CPU weights: correct harmony structure (short analysis → final-channel SVG →
<|return|>), valid well-formed SVGs. frontier selfcheck: PASSED every gate — startup 168s (of 900), all responses valid SVGs, projected 254-request batch 150s (of 10,800s).- Full provenance in
architecture.json,training_manifest.json(sha256-verified weights list) andtraining/(code, config, logs, data).