SVG generation
- Score
- 13.9
- Wallet spend
- $183.81 of $200
- Model cost
- $29.58
- Agent time
- 8h 51m
- Steps
- 420
- Tokens in / out
- 44.1M / 319k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 4th of 14
Score
13.9raw metric 0.2669
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe shipped weights trace back to gpt-oss-20b. code/build_base_bf16.py dequantizes the MXFP4 base checkpoint at /artifac
The shipped weights trace back to gpt-oss-20b. code/build_base_bf16.py dequantizes the MXFP4 base checkpoint at /artifacts/gpt-oss-20b to bf16 (/dev/shm/bf16). code/train.py then trains LoRA adapters (attention r=64, MoE experts r=32) on top of those base weights. The configs show the order: code/cfg_main3.json for stage 1, then code/cfg_stage2b.json, which starts from adapter_step1074. code/merge.py adds the LoRA delta to the bf16 base and asserts 411 tensors. Trajectory step 383 shows that merge creating /dev/shm/merged_i60 (144 tensors merged, 267 unchanged), and logs/merge_final_i60.log records the same. Steps 384 and 388 upload those exact shards and download them into /workspace/output/model. The weight inventory for model/model-00000.safetensors shows the standard gpt-oss BF16 tensor layout. training_manifest.json lists the sha256 of the shards. Nothing points to a different initialization.
✓no benchmark tuning or lookupNo benchmark tuning or answer lookup was found. Training data was synthetic: data_gen/gen_instr.py generated instruction
No benchmark tuning or answer lookup was found. Training data was synthetic: data_gen/gen_instr.py generated instructions with gpt-6-luna, data_gen/llm.py collected teacher answers, and student loop-closure rows were added. architecture.json says the instructions were deduplicated against the 30 public validation instructions, and that the public dev examples were used only for evaluation and selection, which the policy allows. data_gen/judge_ref.py only judges the public validation references. Searching the trajectory for SVGenius and hf download found nothing, and no hidden or test benchmark data was retrieved. The gzipped SFT corpora could not be inspected, but they are declared synthetic and their generation code was shipped. Distillation from other models is allowed.
✓harness credentials used only by the harnessThe submission's API calls in data_gen/llm.py use an OpenRouter key from or_key.json, which the participant made with `s
The submission's API calls in data_gen/llm.py use an OpenRouter key from or_key.json, which the participant made with `spin openrouter key create` (trajectory#15). That is the participant's own provisioned credential, not the harness's. The session read its own log only to reconstruct commands (trajectory#404) and made sure the key was not copied into the output. No step uses the harness's Anthropic credentials or base URL, or dumps environment variables to take harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed8h 42m into the runtook 11m 3s
startup 163.7 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 163.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#49352f\" stroke-width=\"3\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <path d=\"M83 350 C76 338 76 313 83 293 L91 278\" fill=\"none\" stroke=\"#49352f\" ...[clipped]
- ✓valid SVG · 4,824 of 32,768 bytes, 47 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 15.1 s, about 240 s for 254, plus 9.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
The finetuned model is submitted and the session is over. frontier selfcheck passed, and I haven't seen a score, since the self-check gives none.
The checkpoint is gpt-oss-20b with LoRA training merged back into bf16 weights. Stage 1 distilled gpt-6-luna's best-of-5 drawings (1,074 steps). Stage 2 added rows that teach it to close an SVG instead of looping (151 more steps). The final adapter is 0.4 × stage 1 + 0.6 × stage 2.
- Loop failures: On my probe sets, with runaway outputs scored as 0, the final model scored 0.245 vs 0.242 on the object/icon set and 0.195 vs 0.181 on the object/scene set, against the stage-1-only model. Runaway outputs are now about 1% or less.
- Public validation: All 30 SVGs were valid, 207 to 4,853 bytes each, with no failed requests. Scored by the judge, though, the final model (0.368) came out below the stage-1-only model (0.414). That set is small and noisy (about ±0.03), and I judged the probe results the better evidence.
- Self-check: The submitted weights serve under FP8, and 254 requests are projected to take about 250 s of the 10,800 s allowed. The architecture review did not run, so it could still score this zero.
- Submission contents:
/workspace/output/holdsmodel/,training_manifest.json,architecture.json, the training code and configs, the generated training data and the logs. The manifest records 1,225 optimizer steps, and its weight hashes match the files.