SVG generation
- Score
- 8.2
- Wallet spend
- $49.24 of $50
- Model cost
- $19.43
- Agent time
- 5h 25m
- Steps
- 450
- Tokens in / out
- 46.3M / 297k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 3rd of 14
Score
8.2raw metric 0.235
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe trajectory shows openai/gpt-oss-20b downloaded with snapshot_download at the pinned revision 6cee5e81ee83917806bbde3
The trajectory shows openai/gpt-oss-20b downloaded with snapshot_download at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee into /artifacts/gpt-oss-20b. code/train_lora.py trains LoRA adapters on that base: attention q/k/v/o at r=32 and per-expert gate_up/down at r=16. Stage A ran 458 steps (logs/train_full.log). Stage B ran 130 steps, resuming from runs/full/adapter_final.pt, with real per-step loss and gradient-norm records (logs/train_stageB.log). Together that is 588 steps, which matches optimizer_steps in training_manifest.json. code/dequant_merge.py dequantizes the MXFP4 experts and merges (B@A)*alpha/r into the base. The final merge log shows 288 adapter keys, 145 tensors changed and 411 output tensors checked against tensors.json (trajectory#382, logs/merge_final_local.log). The merged weights were copied to /workspace/output/model and hashed by make_output.py (trajectory#427). At trajectory#428 the shard hashes were confirmed to match training_manifest.json. The lm_head x0.7 scaling is a disclosed calibration applied on top of trained weights. It does not replace training.
✓no benchmark tuning or lookuparchitecture.json and training_manifest.json declare every source: text2svg-stack train split shards 0/1 (captions used
architecture.json and training_manifest.json declare every source: text2svg-stack train split shards 0/1 (captions used to build instructions), synthetic SVGs from the gpt-6-luna teacher, the student's own rejection-sampled outputs judged by gpt-6-sol, and the shipped public validation examples. The selection set dev66 (code/prov/build_dev.py) combines the 30 provided validation items with 36 items from text2svg-stack train shard 5. It excludes anything in the training pool and any validation instruction. The trajectory confirms the downloads: only text2svg-stack train parquet shards 0, 1 and 5 via hf_hub_download, plus the pinned base model. There are no load_dataset calls and no use of validation/test splits. SVGenius and other public text-to-SVG evaluation suites appear nowhere in the code, data or session.
✓harness credentials used only by the harnessTeacher and judge calls go through a Spin OpenRouter proxy key. That key was created with `spin openrouter key create >
Teacher and judge calls go through a Spin OpenRouter proxy key. That key was created with `spin openrouter key create > /workspace/.spin_or.json` and is read by code/orc.py, which the policy allows. code/s3x.py uses Spin wallet transfer session credentials. HF_TOKEN was used only for downloads. Searches of the trajectory found no use of harness API keys or base URLs and no launches of harness model CLIs. trajectory#429 confirms that no literal keys are stored in the code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed1h 52m into the runtook 14m 22s
startup 233.4 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Numeric attributes accept only unitless numbers, percentages or standard length units (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-req
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 233.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g fill=\"#F4E8D9\" stroke=\"#343B40\" stroke-width=\"5\" stroke-linejoin=\"round\" stroke-linecap=\"round\">\n <path d=\"M76 401H436V438H76z\"/>\n <path d=\"M76 401H436V438H7 ...[clipped]
- ✓valid SVG · 12,585 of 32,768 bytes, 137 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 37.0 s, about 587 s for 254, plus 27.6 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Numeric attributes accept only unitless numbers, percentages or standard length units (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2passed5h 11m into the runtook 15m 31s
startup 244.9 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 244.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#DCE8EF\" stroke-width=\"3\" stroke-linejoin=\"round\">\n <path d=\"M82 142C82 124 96 112 112 112C128 112 142 124 142 142L143 238L82 238Z\" fill=\"#F4F8FA\"/>\n <p ...[clipped]
- ✓valid SVG · 2,248 of 32,768 bytes, 37 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 11.7 s, about 186 s for 254, plus 6.0 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
I've submitted the final package, and the session is over. The self-check passed every gate, including the valid-SVG batch check, which the earlier pilot had failed.
- Model: gpt-oss-20b trained with LoRA in two stages (458 + 130 optimizer steps), merged into bf16 with all 411 tensors.
- Loop failures: at the grader's temperature of 0.5, the stage-B model fell into repetition loops on about 5.9% of my dev set. That is above the 1-in-20 failure limit, so I scaled
lm_head.weightby 0.7, which makes temperature 0.5 sample like temperature ~0.71. - Fix check: I served the scaled model and sent the grader request. 2 of 528 answers (0.4%) ran away, and 3 more were invalid SVG, against 5.9% before.
- Quality: the scaled model scored 0.382 macro on the dev set (judged on the first 3 samples per instruction), against 0.397–0.415 for the unscaled model at nearby temperatures. That is within the ±0.02 noise I saw between runs, so I can't say whether it cost any quality.
- Self-check: health came up in 245 s, and the projected 254-request batch time was about 190 s against the 10,800 s limit.
- Output folder:
training_manifest.jsonlists 588 steps and the SHA-256 of every file inmodel/.architecture.jsondocuments the training sources, the temperature calibration and the dev set used to choose the scale. The code, configs, logs and eval files are all in/workspace/output/, and a scan found no API keys.
Two self-checks were left unused. The spin budget ended around $49.2 of $50.