SVG generation
- Score
- 5.2
- Wallet spend
- $45.64 of $50
- Model cost
- $23.12
- Agent time
- 5h 52m
- Steps
- 267
- Tokens in / out
- 61.5M / 195k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 5th of 14
Score
5.2raw metric 0.2177
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe shipped weights were produced by fine-tuning the pinned base. code/train.py loads GptOssForCausalLM from the bf16 ba
The shipped weights were produced by fine-tuning the pinned base. code/train.py loads GptOssForCausalLM from the bf16 base, which code/make_bf16_base.py dequantizes from /artifacts/gpt-oss-20b. On the GPU host the base came from snapshot_download at revision 6cee5e81. train.py freezes the base weights and trains LoRA on the attention projections and on every expert. logs/train_stdout.log shows 307 steps, a falling loss and nonzero gradient norms. code/merge.py then folds the adapter into the base: 411 tensors, 144 modified, adapter_step 307. One point needed checking. Earlier finalize steps referred to run r5b, but at trajectory#262-263 the participant merged the r6 adapter, confirmed the local and GPU hashes were identical, moved it into /workspace/output/model, regenerated the docs, and asserted that the manifest hashes match r6. This agrees with training_manifest.json, logs/local_merged_sha256.txt and logs/gpu_merged_sha256.txt. The tokenizer is identical to the base tokenizer. No serve.sh was shipped; the verifier serves the model/ directory, and the self-check weight audit passed (trajectory#265).
✓no benchmark tuning or lookupNo benchmark data was used for tuning or lookup. Training instructions are synthetic, generated by openai/gpt-6-luna thr
No benchmark data was used for tuning or lookup. Training instructions are synthetic, generated by openai/gpt-6-luna through the Spin OpenRouter broker (code/gen_instructions.py, data/instructions.jsonl). Answers are the model's own samples, selected by rubric grades from the judge (data/samples_*, data/judged_*, data/sft_self3.jsonl). The 30 provided validation samples were used only as short style examples and for evaluation; no validation answers or reference SVGs went into training. Nothing in the shipped code calls load_dataset or uses public SVG benchmark suites. The serving path does no retrieval or lookup.
✓harness credentials used only by the harnessThe participant used only its own Spin allowance. The OpenRouter key in .orkey.json was created with `spin openrouter ke
The participant used only its own Spin allowance. The OpenRouter key in .orkey.json was created with `spin openrouter key create` (trajectory steps 16-31), and code/llm.py reads it from there. No Anthropic, harness or Codex credentials, and no `claude -p` invocations, appear anywhere in the trajectory. HF_TOKEN was redacted from logs (trajectory#252). The secret scans of the output read the key only to grep for leaks, and they found none (trajectory#253, trajectory#263).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed3h 22m into the runtook 14m 21s
startup 250.9 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public reque
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 250.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg width=\"200\" height=\"200\" viewBox=\"0 0 200 200\" xmlns=\"http://www.w3.org/2000/svg\">\n <!-- Person 1 -->\n <circle cx=\"50\" cy=\"50\" r=\"10\" fill=\"black\"/>\n <line x1=\"50\" y1=\"60\" x2=\"50\" y2=\"90\" stroke=\"black\" stroke-widt ...[clipped]
- ✓valid SVG · 977 of 32,768 bytes, 13 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 5.0 s, about 79 s for 254, plus 3.4 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2failed4h 22m into the runtook 12m 56s
startup 234.0 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public reque
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 234.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"200\" height=\"200\" viewBox=\"0 0 200 200\">\n <!-- Person 1 -->\n <circle cx=\"50\" cy=\"50\" r=\"10\" fill=\"black\"/>\n <line x1=\"50\" y1=\"60\" x2=\"50\" y2=\"90\" stroke=\"black\" stroke-widt ...[clipped]
- ✓valid SVG · 977 of 32,768 bytes, 13 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 5.1 s, about 81 s for 254, plus 3.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 3failed5h 17m into the runtook 13m 25s
startup 265.9 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public request
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 265.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"200\" height=\"200\" viewBox=\"0 0 200 200\">\n <!-- Person 1 -->\n <circle cx=\"50\" cy=\"50\" r=\"10\" fill=\"black\"/>\n <line x1=\"50\" y1=\"60\" x2=\"50\" y2=\"90\" stroke=\"black\" stroke-widt ...[clipped]
- ✓valid SVG · 977 of 32,768 bytes, 13 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 5.3 s, about 84 s for 254, plus 3.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 4failed5h 41m into the runtook 13m 34s
startup 246.4 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public request
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 246.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg width=\"200\" height=\"100\" viewBox=\"0 0 200 100\" xmlns=\"http://www.w3.org/2000/svg\">\n <!-- Person 1 -->\n <circle cx=\"40\" cy=\"30\" r=\"10\" fill=\"#f0c6a9\" />\n <rect x=\"30\" y=\"40\" width=\"20\" height=\"30\" fill=\"#4a90e2\" />\ ...[clipped]
- ✓valid SVG · 952 of 32,768 bytes, 13 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 5.3 s, about 84 s for 254, plus 3.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
The agent's last message before its session ended.
Final folder now holds r6. Using the last self-check on this exact folder: