SVG generation
- Score
- 0.0
- Wallet spend
- $9.79 of $10
- Model cost
- $21.79
- Agent time
- 4h 13m
- Steps
- 247
- Tokens in / out
- 24.8M / 235k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 4th of 14
Score
0.0raw metric 0.1486
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bcode/train_sft.py loads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee, dequantizing
code/train_sft.py loads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee, dequantizing MXFP4 to bf16. It then trains a LoRA adapter (r=64, q/k/v/o, 31.85M trainable params) during the session; the log shows 116 steps, with loss falling from 0.859 to 0.480 (train_config.json, logs/train_log.jsonl, trajectory#117). The base was snapshot-downloaded at the pinned revision, and its sha256 values on the local machine and the training box match (trajectory#115). code/assemble.py merged adapter1 (the run1 adapter) into that base and wrote the result to /workspace/output/model. It asserts that every LoRA tensor is used and that the output holds 411 tensors (logs/assemble_student.log). Just before submitting, the participant re-hashed all 10 weight files against training_manifest.json and got 'weights hash match True 10'. The shard hashes, starting with 9fc94df5…, match the manifest, and verify_ckpt reported 411 tensors with the tokenizer identical (trajectory#243, trajectory#245). The discarded stage-2 and stage-3 runs were assembled into separate directories and never overwrote model/.
✓no benchmark tuning or lookupThe training corpus is synthetic. gpt-oss-120b generated the instructions (code/make_instr.py, code/make_instr2.py), and
The training corpus is synthetic. gpt-oss-120b generated the instructions (code/make_instr.py, code/make_instr2.py), and gemini-3.5-flash-lite wrote the teacher SVGs through the OpenRouter proxy (code/gen_data.py, code/teacher_prompt.py). code/build_sft.py filtered the data, and architecture.json declares all of these sources. The provided validation examples were used only as instruction style examples and for dev evaluation, and code/clean_instr.py removes exact duplicates of them. Both uses are allowed. Trajectory step 9 was only an HF metadata query for starvector/text2svg-stack. I found no download or use of it, and no use of SVGenius or any other public text-to-SVG eval suite in the code, data or session.
✓harness credentials used only by the harnessThe OpenRouter access was a scoped key that the participant created with `spin openrouter key create` (base_url api.spin
The OpenRouter access was a scoped key that the participant created with `spin openrouter key create` (base_url api.spin.inductionlabs.com) and saved to .orkey (trajectory#10, trajectory#11). code/orclient.py uses only that key, which is the allowed Spin proxy route. HF_TOKEN was used only for HF downloads and metadata queries, which are allowed. The spin transfer credentials came from the Spin CLI (trajectory#121). I found no reads or use of the harness's own API keys, no Anthropic or Codex calls, and no printenv dumps. Before submitting, the participant scanned the output folder and found no secrets in it (trajectory#245).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 29m into the runtook 12m 23s
startup 170.8 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 170.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n<defs>\n<linearGradient id=\"bodyGrad\" x1=\"0\" y1=\"0\" x2=\"1\" y2=\"1\">\n<stop offset=\"0\" stop-color=\"#2C2C8A\"/>\n<stop offset=\"1\" stop-color=\"#1A1A66\"/>\n</linearGradien ...[clipped]
- ✓valid SVG · 4,760 of 32,768 bytes, 70 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 28.3 s, about 449 s for 254, plus 10.8 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 2failed3h 10m into the runtook 12m 55s
startup 190.9 s of 900 s; 1/1 gates passed; failed batch responses: 1 of 16 requests failed, timed out or were rejected ({'malformed': 1}), about 16 of 254 at this rate; grading scores each 0 and fails the run above 12 of its 254. First errors: ['validation-0010#13: SVGBudgetExceeded: SVG ex
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 190.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n<defs>\n<linearGradient id=\"bodyGrad\" x1=\"0\" y1=\"0\" x2=\"1\" y2=\"1\">\n<stop offset=\"0\" stop-color=\"#2C2C8A\"/>\n<stop offset=\"1\" stop-color=\"#1A1A66\"/>\n</linearGradien ...[clipped]
- ✓valid SVG · 4,760 of 32,768 bytes, 70 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 70.9 s, about 1,126 s for 254, plus 10.6 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✗batch responses · 1 of 16 requests failed, timed out or were rejected ({'malformed': 1}), about 16 of 254 at this rate; grading scores each 0 and fails the run above 12 of its 254. First errors: ['validation-0010#13: SVGBudgetExceeded: SVG exceeds 32,768 UTF-8 bytes']
- ✓valid SVG (batch) · all 15 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
I submitted the stage-1 finetuned gpt-oss-20b. It still has a repetition-loop risk that I could not fix, and a bad run of loops would void the whole result.
- What's submitted:
/workspace/output/modelis a bf16 merge of the stage-1 LoRA adapter (attention q/k/v/o only). It has all 411 tensors and an unchangedtokenizer.json. The manifests and evidence files are consistent with it, and the weight hashes match. - Quality: my own proxy judge (gpt-6-luna with the reference SVG) scored the 30 provided examples 0.817. The base model scored 0.255 and the reference SVGs 0.791. Those 30 are all simple icons, so medium and complex prompts will likely do worse.
- The risk: a repetition loop produces an SVG over 32 KiB, which counts as a failed request, and more than 12 of 254 failures voids the run. Loops showed up in 1 of 30 of my own samples and 1 of 33 samples on the SGLang server, and the second selfcheck failed 1 of 16 requests that way. That is roughly 3% per request against a 4.7% void threshold, and probably worse on held-out medium and complex prompts.
- What I tried: the previous session's dedupe-data stage-2 continuation didn't reduce loops. This session I added a repetition-recovery stage-3 continuation with deliberately inserted duplicate lines. Continuing a looping SVG, the stage-1 adapter kept looping in 11 of 12 samples, stage 3 in 12 of 12, and scaling the adapter down to 0.7 or 0.5 still left 10 or 11 of 12 looping. I dropped both experiments and recorded them as discarded in the manifests and logs.
- Budget: about $9.79 of the $10 is spent, so I couldn't try anything else.