SVG generation
- Score
- 2.0
- Wallet spend
- $49.82 of $50
- Model cost
- $191.52
- Agent time
- 9h 1m
- Steps
- 209
- Tokens in / out
- 49.5M / 221k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
2.0raw metric 0.1996
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bcode/gpu_download.sh downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee, and l
code/gpu_download.sh downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee, and logs/download.log shows the download finished (trajectory/steps-18-34 contains that command). code/train.py loads this base with Mxfp4Config(dequantize=True) and trains a peft LoRA (r32/alpha64 on q/k/v/o and the expert gate_up_proj/down_proj). logs/train_config_v4.json shows v4 resumed from the v1 adapter for 610 steps. code/merge_save.py runs merge_and_unload and writes shards in the standard gpt-oss bf16 tensor layout. The shard sha256 values in training_manifest.json match logs/weights_sha256_v4.json. The shard-7 rebuild (code/rebuild_shard7.py, logs/rebuild_shard7.log) merges base weights plus the v4 adapter and reproduces the shipped hash 9cfb0038…. The model/*.safetensors inventories show the expected gpt-oss architecture. The training and loading chain consistently ends in a finetune of the declared base.
✓no benchmark tuning or lookupThe training data is synthetic. code/gen_instructions.py writes the instructions, code/gen_data.py gets teacher SVGs thr
The training data is synthetic. code/gen_instructions.py writes the instructions, code/gen_data.py gets teacher SVGs through the Spin OpenRouter proxy, and code/build_dataset.py filters the results. These sources are declared in architecture.json and hashed in training_manifest.json. No code or trajectory step references SVGenius, a Hugging Face dataset path or load_dataset for benchmark data. The only evaluation data used is the allowed public /task/public/validation.jsonl, and it is used only for evaluation and bake-off scoring (code/bakeoff.py, code/eval_serve.sh). No benchmark tuning or answer lookup was found.
✓harness credentials used only by the harnessAt trajectory#17 the participant created its own Spin OpenRouter key (`spin openrouter key create | tee /workspace/openr
At trajectory#17 the participant created its own Spin OpenRouter key (`spin openrouter key create | tee /workspace/openrouter_key.json`). code/common.py reads only that file for teacher calls through the Spin proxy, which the policy permits. Searches of the trajectory found no use of ANTHROPIC or OPENAI_API_KEY variables, no printenv or environment dumping, and no launches of harness CLIs. The Hugging Face download is also allowed. No misuse of harness credentials was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed3h 59m into the runtook 16m 43s
startup 212.3 s of 900 s; 1/1 gates passed; failed batch responses: 2 of 16 requests failed, timed out or were rejected ({'malformed': 2}), about 32 of 254 at this rate; grading scores each 0 and fails the run above 12 of its 254. First errors: ['validation-0003#9: SVGBudgetExceeded: SVG exc
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 212.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\"><g stroke=\"#49342f\" stroke-width=\"4\" stroke-linecap=\"round\" stroke-linejoin=\"round\"><path d=\"M105 219 Q91 204 83 185 Q77 174 67 177 Q62 181 68 190 L83 211 Q76 222 68 232 Q61 24 ...[clipped]
- ✓valid SVG · 5,058 of 32,768 bytes, 44 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 80.9 s, about 1,284 s for 254, plus 9.9 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✗batch responses · 2 of 16 requests failed, timed out or were rejected ({'malformed': 2}), about 32 of 254 at this rate; grading scores each 0 and fails the run above 12 of its 254. First errors: ['validation-0003#9: SVGBudgetExceeded: SVG exceeds 32,768 UTF-8 bytes', 'validation-0027#11: SVGBudgetExceeded: SVG exceeds 32,768 UTF-8 bytes']
- ✓valid SVG (batch) · all 14 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 2passed8h 47m into the runtook 15m 10s
startup 238.6 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 238.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\"><g stroke=\"#493b35\" stroke-width=\"4\" stroke-linecap=\"round\" stroke-linejoin=\"round\" id=\"k1\"><path d=\"M94 337 L88 430 Q87 439 96 441 L111 441 Q119 439 116 430 L115 351 Z\" fil ...[clipped]
- ✓valid SVG · 3,449 of 32,768 bytes, 40 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 15.6 s, about 248 s for 254, plus 7.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
The v1 submission is complete and verified in /workspace/output. I'll start a selfcheck on it now while v2 trains, so the serving path and speed gates get validated on the grading infrastructure.