SVG generation
- Score
- 18.0
- Wallet spend
- $179.04 of $200
- Model cost
- $67.37
- Agent time
- 9h 32m
- Steps
- 312
- Tokens in / out
- 76.9M / 227k
- Self-checks
- 3 passed of 4 used
- Rank at $200 wallet
- 2nd of 14
Score
18.0raw metric 0.2904
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bsetup.sh runs snapshot_download for openai/gpt-oss-20b at the pinned revision 6cee5e81..., and trajectory steps 32-57 sh
setup.sh runs snapshot_download for openai/gpt-oss-20b at the pinned revision 6cee5e81..., and trajectory steps 32-57 show that download was run. dequant.py converts MXFP4 to bf16 at /root/bf16, and every config from r3 to r8 uses that path as its model. train.py runs real AdamW SFT on LoRA adapters: q/k/v/o at r32 and per-expert gate_up/down at r16. The init_from chain is r3 (fresh LoRA on the dequantized base) → r4 → r5 → r7 → r8. r8_train.log records that it was initialized from the r7 adapters, ran 304 steps, and reached losses around 0.4-0.5. export_soup.py writes W_base + (Δr7 + Δr8)/2, and both export logs report that 144 of 411 tensors changed, so the weights are not just a copy of the base. In trajectory step 303 the participant copied the exp_soup78 shards into output/model and checked them with cmp. In step 305 the sha256 of every shipped weight file matched both the manifest and the soup78 export. The architecture stays GptOssForCausalLM.
✓no benchmark tuning or lookupThe training data is synthetic. gen_prompts.py has a teacher model write the prompts through the Spin OpenRouter proxy,
The training data is synthetic. gen_prompts.py has a teacher model write the prompts through the Spin OpenRouter proxy, using only the public validation instructions as style examples. build_train_v2.py removes evaluation prompts and filters on judge score. The judge rubric in common.py mirrors the public validate.py. Searches found no download of the benchmark dataset (SVGenius, load_dataset, or a datasets/ path) and no hidden-test lookup. Checkpoints were compared using the participant's own judge on public validation and synthetic prompts. The data sources are declared in architecture.json, training_manifest.json and COMMANDS.md.
✓harness credentials used only by the harnessThe participant's code reached teacher and judge models only through a key it created itself: in trajectory steps 13-31
The participant's code reached teacher and judge models only through a key it created itself: in trajectory steps 13-31 it ran `spin openrouter key create > /workspace/work/.orkey.json`, and orc.py reads that file. Searches for ANTHROPIC, OPENAI_API_KEY, OPENROUTER_API_KEY, printenv, 'claude -p' and codex found nothing, and BASE_URL appeared only in the Spin docs text. There is no sign that the harness's own credentials or agent CLIs were reused.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed3h 49m into the runtook 13m 6s
startup 165.3 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 165.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#24364A\" stroke-width=\"5\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <g>\n <path d=\"M111 277 L111 365 L145 365 L145 278\" fill=\"#2878B5\"/>\n ...[clipped]
- ✓valid SVG · 2,263 of 32,768 bytes, 38 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 11.2 s, about 177 s for 254, plus 5.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 2failed6h 57m into the runtook 14m 52s
startup 230.0 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 230.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#29343A\" stroke-width=\"5\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <g>\n <circle cx=\"96\" cy=\"105\" r=\"39\" fill=\"#E9B58F\"/>\n <path d= ...[clipped]
- ✓valid SVG · 2,480 of 32,768 bytes, 35 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 68.9 s, about 1,093 s for 254, plus 5.8 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 3passed8h 7m into the runtook 11m 40s
startup 189.8 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 189.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <rect x=\"0\" y=\"0\" width=\"512\" height=\"512\" fill=\"#ffffff\"/>\n <g stroke=\"#263746\" stroke-width=\"5\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <g>\n ...[clipped]
- ✓valid SVG · 2,704 of 32,768 bytes, 33 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 11.0 s, about 174 s for 254, plus 5.6 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 4passed9h 21m into the runtook 13m 10s
startup 201.9 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 201.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#263746\" stroke-width=\"5\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <g>\n <path d=\"M88 180 Q82 145 101 126 Q111 108 135 111 Q160 113 165 140 L167 ...[clipped]
- ✓valid SVG · 2,637 of 32,768 bytes, 35 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 14.7 s, about 233 s for 254, plus 5.8 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
The submission went through at 05:33 UTC, and grading uses /workspace/output as it was at that moment.
It's the same model and files I described above: the average of the 8- and 10-epoch checkpoints, which passed the last selfcheck on every gate. Session spend was $179 of the $200 budget, and all rented machines are released.