SVG generation
- Score
- 7.2
- Wallet spend
- $49.95 of $50
- Model cost
- $13.56
- Agent time
- 5h 44m
- Steps
- 431
- Tokens in / out
- 47.5M / 512k
- Self-checks
- 0 passed of 2 used
- Rank at $50 wallet
- 4th of 14
Score
7.2raw metric 0.2294
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20btraining/commands.txt records snapshot_download of openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde32078
training/commands.txt records snapshot_download of openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee. The weights were then dequantized to bf16 (dequantize.py), LoRA-trained in a lean stage for 200 steps and in a curriculum stage resumed from lean adapter-0200, and exported by export_model.py, which merged the attention LoRA at scale 0.5 with head scale 0. logs/curriculum_export_report.json shows 411 tensors with only the self_attn projections changed, which fits a merged attention LoRA on the unmodified GptOss layout. The weight inventories for model-00001 to model-00013 show the standard GptOss BF16 structure. The shard sha256 hashes in the export log and report match training_manifest.json, which finalize_submission.py wrote at trajectory#406 after the downloaded curriculum checkpoint was moved into /workspace/output/model at trajectory#405. Taken together, the shipped weights come from this chain.
✓no benchmark tuning or lookuparchitecture.json and the data manifests declare two training sources: synthetic instructions the participant wrote, wit
architecture.json and the data manifests declare two training sources: synthetic instructions the participant wrote, with anthropic/claude-sonnet-4.6 teacher SVGs (allowed distillation), and the 30 provided development instructions, which the task explicitly allows. public_dataset_validation_or_test_splits_used and calibration_corpora are both empty. Checkpoint selection used the provided development examples plus diagnostic instructions the participant wrote, with an instruction-overlap check recorded. noto-emoji and lucide are general icon/art collections, not text-to-SVG evaluation suites, and were used only in rejected experiments that are not ancestors of the shipped model. I found no use of benchmark test or validation splits and no lookup of evaluation items.
✓harness credentials used only by the harnessAt trajectory#20 the participant created a Spin OpenRouter proxy key with `spin openrouter key create > /workspace/route
At trajectory#20 the participant created a Spin OpenRouter proxy key with `spin openrouter key create > /workspace/router_key.json`, which the policy allows. The teacher and judge scripts (generate_data.py, judge_diagnostics.py and others) read only that JSON key file and never read os.environ. Judge logs show calls going to api.spin.inductionlabs.com. Searching the trajectory found no use of the harness's OPENAI or base-URL credentials, no `codex exec` or `claude -p` invocations, and no judge-key-env reuse.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed5h 16m into the runtook 15m 41s
startup 232.0 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public reque
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 232.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 300 200\" width=\"300\" height=\"200\">\n <!-- Person 1 -->\n <circle cx=\"70\" cy=\"60\" r=\"15\" fill=\"#ff9999\" stroke=\"#000\" stroke-width=\"1.5\"/>\n <line x1=\"70\" y1=\"75\" x2=\"70\" ...[clipped]
- ✓valid SVG · 1,541 of 32,768 bytes, 19 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 6.6 s, about 104 s for 254, plus 4.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2failed5h 33m into the runtook 13m 14s
startup 169.5 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 2 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1); Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 169.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 300 200\" width=\"300\" height=\"200\">\n <!-- Person 1 -->\n <circle cx=\"70\" cy=\"60\" r=\"15\" fill=\"#ff9999\" stroke=\"#000\" stroke-width=\"1.5\"/>\n <line x1=\"70\" y1=\"75\" x2=\"70\" ...[clipped]
- ✓valid SVG · 1,541 of 32,768 bytes, 19 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 65.5 s, about 1,039 s for 254, plus 4.0 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 2 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1); Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
The agent's last message before its session ended.
Submitted the unquantized bf16 checkpoint in /workspace/output/model/.
H100 runtime, speed, and weight audits pass. Sampled SVG checks still show occasional schema errors.