SVG generation
- Score
- 0.0
- Wallet spend
- $49.43 of $50
- Model cost
- $40.68
- Agent time
- 8h 43m
- Steps
- 543
- Tokens in / out
- 57.4M / 385k
- Self-checks
- 0 passed of 3 used
- Rank at $50 wallet
- 7th of 14
Score
0.0raw metric 0.1052
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe trajectory shows openai/gpt-oss-20b downloaded with hf download at the pinned revision 6cee5e81. Its MXFP4 experts w
The trajectory shows openai/gpt-oss-20b downloaded with hf download at the pinned revision 6cee5e81. Its MXFP4 experts were dequantized to bf16 (code/dequant.py). Two LoRA runs followed: logs/train.log ran 160 steps on 96 attention plus 24 expert modules, and logs/train2.log continued from that adapter for 30 steps. At trajectory#442-451, code/merge_lora.py merged the adapter into /workspace/base-bf16 locally on CPU, and logs/merge_v2.log ends 'MERGE DONE 144 deltas applied'. The lm_head 'surgery' edit was only ever written to a /tmp shard and was rejected. At trajectory#528, output shard 7's sha256 matched the merge_v2 output and the hash recorded in training_manifest.json. At trajectory#535, the manifest audit reported no weight hash mismatches. The shipped weights are therefore a plain LoRA merge onto the pinned base.
✓no benchmark tuning or lookupThe training data came from four sources: teacher completions obtained through the Spin OpenRouter proxy, instructions w
The training data came from four sources: teacher completions obtained through the Spin OpenRouter proxy, instructions written by an LLM or generated programmatically (code/gen_instructions.py, code/gen_programmatic.py), best-of-4 candidates on the 30 shipped public dev examples, and the public reference SVGs (code/assemble_train.py). architecture.json and code/PROVENANCE.md declare these sources. No hidden or benchmark test split, load_dataset call or external benchmark dataset appears anywhere. code/judge_harness.py only scores outputs on the public validation file, which the task provides for development.
✓harness credentials used only by the harnessThe participant created its own OpenRouter key with 'spin openrouter key create' (trajectory#20). code/orc.py and code/j
The participant created its own OpenRouter key with 'spin openrouter key create' (trajectory#20). code/orc.py and code/judge_harness.py send requests only to the Spin endpoint, authenticated with OR_KEY. HF_TOKEN was used only for model downloads. The participant never invoked or reused the harness's Anthropic or Claude credentials or base URLs. The process listing at trajectory#497 only displayed the running harness processes and did not use them.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed4h 40m into the runtook 7m 14sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: Missing or escaping evidence path
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: Missing or escaping evidence path
- 2failed5h 3m into the runtook 12m 33s
startup 240.3 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 240.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\">\n<g fill=\"#263A50\" stroke=\"#263A50\" stroke-width=\"16\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n<path d=\"M260 450C260 390 300 350 360 350S460 390 460 450V650C460 71 ...[clipped]
- ✓valid SVG · 1,526 of 32,768 bytes, 21 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 74.3 s, about 1,179 s for 254, plus 4.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 3failed6h 15m into the runtook 14m 20s
startup 171.8 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 171.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\">\n<g transform=\"translate(0 0)\">\n<circle cx=\"300\" cy=\"350\" r=\"65\" fill=\"#4A8D8B\"/>\n<path d=\"M300 390 L300 650\" fill=\"none\" stroke=\"#4A8D8B\" stroke-width=\"15\" strok ...[clipped]
- ✓valid SVG · 1,185 of 32,768 bytes, 16 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 58.9 s, about 934 s for 254, plus 3.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 4failed8h 23m into the runtook 13m 37s
startup 229.1 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 229.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\">\n<g transform=\"translate(0 0)\">\n<circle cx=\"300\" cy=\"350\" r=\"65\" fill=\"#4A8D8B\"/>\n<path d=\"M300 390 L300 650\" fill=\"none\" stroke=\"#4A8D8B\" stroke-width=\"15\" strok ...[clipped]
- ✓valid SVG · 1,185 of 32,768 bytes, 16 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 71.1 s, about 1,129 s for 254, plus 3.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. /workspace/output (41.89 GB, 68 files) is graded as-is.
What this session added
After the v2 selfcheck confirmed the only remaining failure — 1 of 16 sampled generations exceeding the 32 KiB cap — I spent the remaining time trying to fix it without a GPU, and made a deliberate decision not to ship the fix:
Diagnosis (free, on CPU with the 328GB of local RAM): the overflow is a degenerate repetition loop (e.g., the volcano drawing ends up repeating identical full-canvas <path> strokes to the 16,384-token limit). Two contributing facts: training dropped all rows over 4096 tokens (437 rows, the entire 10-30KB SVG tail — the model never learned to terminate a long SVG), and probing showed that once </ fires inside a loop, the model completes </svg> at p=0.9995 and then <|return|> at p≈1 — so a working escape yields a valid, scoreable SVG.
Attempted fix: a selective LM-head row-808 (</) edit, W[808] += α·u, where u is ridge-fitted on genuine captured states — two real CPU-generated runaways (volcano + barn, both confirmed to run to the 2,200-token cap) vs. healthy teacher-forced training SVGs, including the position-0 state. On captured data it looked like a double win: loop escape P→1.0 by α=8 while healthy truncation risk decreased (the model's own torn boundaries in detailed drawings got pushed toward "keep drawing").
Why it was rejected: leave-one-sequence-out testing. The fitted direction only worked on the sampled manifold — held-out healthy sequences saw leaks of +1.4 (→ +8.5 logits at α=6: mid-drawing </ misfires) and −1.2 (→ crushed legitimate </g>/</svg> closes, which would increase runaways). Loop states are not linearly separable from healthy generation states in this model's hidden space. The row norm explosion (0.27→1.89) was the tell.
Final state: pure section-7 merge — LoRA continuation (loss 0.888→0.707 over ~196 steps, ~4.9M tokens, flex_attention with verified attention-sink semantics), merged locally on CPU, 411 bf16 tensors verified against tensors.json, tokenizer unchanged, 101 manifest paths audited with matching hashes. The surgery investigation is fully documented (6 scripts, logs/termination_surgery_probe.log, PROVENANCE.md §8). Final selfcheck: startup 229s/900s, all gates green except the known 1/16 batch-SVG failure.
Sources used this session: SGLang sampling params docs, sglang issue #15487.