SVG generation
- Score
- 0.0
- Wallet spend
- $35.20 of $200
- Model cost
- $2.07
- Agent time
- 5h 52m
- Steps
- 1,281
- Tokens in / out
- 176.0M / 412k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 6th of 14
Score
0.0raw metric 0.1197
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe base is the pinned gpt-oss-20b. data/base_artifact_manifest.json and base_hash.log record the official file sizes, h
The base is the pinned gpt-oss-20b. data/base_artifact_manifest.json and base_hash.log record the official file sizes, hashes and revision 6cee5e81. The trajectory shows the provided /workspace/artifacts/gpt-oss-20b being uploaded with spin s5cmd (trajectory#40) and copied to /workspace/gptoss_base on the H100 (trajectory#46, trajectory#47), and the declared training command uses that directory. train.py loads the base with Mxfp4Config(dequantize=True), trains LoRA (r16/alpha32 on attention projections and lm_head, AdamW) with real gradient steps, then runs merge_and_unload and writes bf16 shards that match the schema. training.log shows 289 examples over 111 steps with loss falling from about 1.03 to 0.85. It also reports tensors that changed versus the base. The shard sha256 values and sizes it records match training_manifest.json. The model shard inventory holds BF16 tensors under the expected gpt-oss names and shapes. No other checkpoint or hand-edit-only path was found.
✓no benchmark tuning or lookuparchitecture.json declares the training sources: the 30 shipped /task/public/validation.jsonl examples plus 259 syntheti
architecture.json declares the training sources: the 30 shipped /task/public/validation.jsonl examples plus 259 synthetic prompts, with SVG targets distilled from teacher openai/gpt-6-sol via the Spin OpenRouter proxy. It states that no public eval suites were used and that selection used only the 30 validation samples. The instruction lists in make_training_data.py and make_complex_training_data.py are hand-written prompts sent to the teacher, and teacher distillation is allowed. Searches of the evidence and trajectory for SVGenius, StarVector, load_dataset and dataset downloads found no benchmark data being fetched or used. The only text-to-SVG matches are the task's own instruction not to train on eval suites. No checkpoint tuned for a benchmark was used.
✓harness credentials used only by the harnessThe teacher calls read /tmp/svg-spin-openrouter.json. The participant inspected spin openrouter key create (trajectory#1
The teacher calls read /tmp/svg-spin-openrouter.json. The participant inspected spin openrouter key create (trajectory#18, trajectory#740), and trajectory#740 shows the file contains base_url https://api.spin.inductionlabs.com/v1 plus a key that was deliberately not printed. That makes it a Spin OpenRouter proxy credential, which is allowed. Searches for OPENAI_API_KEY, OPENAI_BASE_URL, printenv, env dumps and codex exec found no use of harness API keys or base URLs in participant code, no echoing of credentials, and no extra harness CLI launches. The other credentialed activity was Spin compute and s5cmd use, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 44m into the runtook 14m 42s
startup 257.9 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 257.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <rect width=\"512\" height=\"512\" fill=\"#f3f8f9\"/>\n <g fill=\"#fff\" stroke=\"#2e7d8c\" stroke-width=\"4\" stroke-linejoin=\"round\">\n <path d=\"M106 172c0-28 23-51 51-51s5 ...[clipped]
- ✓valid SVG · 2,466 of 32,768 bytes, 38 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 69.3 s, about 1,099 s for 254, plus 6.4 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2passed3h 2m into the runtook 13m 42s
startup 199.8 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 199.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <rect width=\"512\" height=\"512\" fill=\"#ffffff\"/>\n <g fill=\"#ffffff\">\n <circle cx=\"112\" cy=\"112\" r=\"18\"/>\n <circle cx=\"136\" cy=\"112\" r=\"18\"/>\n <circl ...[clipped]
- ✓valid SVG · 9,927 of 32,768 bytes, 258 elements, rendered by rsvg-convert version 2.54.7; the rendering is one flat color
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 45.2 s, about 718 s for 254, plus 15.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 3failed5h 20m into the runtook 14m 42s
startup 238.2 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 238.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <rect width=\"512\" height=\"512\" fill=\"#ffffff\"/>\n <g fill=\"#f0f0f0\" stroke=\"#ffffff\" stroke-width=\"4\" stroke-linejoin=\"round\">\n <circle cx=\"126\" cy=\"126\" r=\" ...[clipped]
- ✓valid SVG · 1,042 of 32,768 bytes, 18 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 80.2 s, about 1,274 s for 254, plus 3.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 4passed5h 38m into the runtook 15m 19s
startup 255.1 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 255.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <rect width=\"512\" height=\"512\" fill=\"#ffffff\"/>\n <g fill=\"#ffffff\">\n <circle cx=\"112\" cy=\"112\" r=\"18\"/>\n <circle cx=\"136\" cy=\"112\" r=\"18\"/>\n <circl ...[clipped]
- ✓valid SVG · 9,927 of 32,768 bytes, 258 elements, rendered by rsvg-convert version 2.54.7; the rendering is one flat color
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 46.5 s, about 738 s for 254, plus 15.9 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-09 12:54:45 UTC. The checkpoint in model/ contains all 411 expected tensors in unquantized BF16.
The final EOS16, three-epoch model passed frontier selfcheck: all 16 responses were schema-valid, startup took 255 seconds, and projected batch time was 754 seconds. On the 30 user-provided examples, its development score was 0.2197. EOS8 scored slightly higher, but failed selfcheck on one malformed SVG, so I submitted the passing checkpoint.