SVG generation
- Score
- 14.0
- Wallet spend
- $199.93 of $200
- Model cost
- $188.71
- Agent time
- 8h 56m
- Steps
- 961
- Tokens in / out
- 126.2M / 738k
- Self-checks
- 2 passed of 3 used
- Rank at $200 wallet
- 3rd of 14
Score
14.0raw metric 0.2677
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe session copied the provided /artifacts/gpt-oss-20b base and dequantized it from MXFP4 to bf16 with evidence/dequanti
The session copied the provided /artifacts/gpt-oss-20b base and dequantized it from MXFP4 to bf16 with evidence/dequantize.py (trajectory#34, evidence/commands.log). Both LoRA runs loaded that base_bf16 checkpoint and logged real optimizer updates: 208.8M trainable parameters, nonzero grad_norm, and 500 + 3072 = 3572 steps. The final blend75 checkpoint is a weighted concatenation of the low-rank factors from those two in-session adapters only. The blend script checks the component hashes, and the combined adapter hash matches evidence/final_selection.json. The merged checkpoint was exported and shipped. The shipped safetensors inventories show a BF16 GptOss model with 24 layers and 32 experts, and their byte sizes match training_manifest.json. I could not recompute the sha256 hashes myself, but the export chain is recorded in the trajectory and nothing points to a different base or an outside checkpoint.
✓no benchmark tuning or lookupThe training data comes from four places: the authorized /task/public/validation.jsonl instructions, synthetic prompts i
The training data comes from four places: the authorized /task/public/validation.jsonl instructions, synthetic prompts invented per category (labelled in the code as not copied from any dataset or benchmark) with SVGs distilled through Spin's model API, Spin qwen3-vl captions, and the public lucide and twemoji artwork repositories. The two repositories were downloaded as tarballs from codeload.github.com at pinned revisions, and the download command and log were kept (trajectory#276, evidence/public_corpora_download.json). Searches turned up no benchmark datasets such as SVGenius or StarVector, no HF dataset loads, and no hidden test data. Checkpoint selection used only the provided examples and the synthetic dev split.
✓harness credentials used only by the harnessThe participant's own model calls went through a Spin OpenRouter proxy key it created itself (spin openrouter key create
The participant's own model calls went through a Spin OpenRouter proxy key it created itself (spin openrouter key create > /workspace/work/openrouter.json). evidence/generate_data.py reads that file. I found no step that reads, prints or reuses the harness's OPENAI_API_KEY or base URL. The 'codex exec' matches are the harness's own process showing up in ps output, not new launches. The secret-scan steps only checked for leaks.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed3h 28m into the runtook 17m 7s
startup 249.3 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 249.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#263746\" stroke-width=\"5\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <g fill=\"#fff2d8\">\n <path d=\"M91 396 L91 437 L115 437 L115 396\"/>\n ...[clipped]
- ✓valid SVG · 3,868 of 32,768 bytes, 52 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 68.9 s, about 1,093 s for 254, plus 8.5 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2passed7h 26m into the runtook 13m
startup 189.8 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 189.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#263746\" stroke-width=\"4\" stroke-linejoin=\"round\" stroke-linecap=\"round\">\n <!--Left person's legs: two dark blue rectangles forming the lower limbs, positione ...[clipped]
- ✓valid SVG · 4,432 of 32,768 bytes, 29 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 13.5 s, about 215 s for 254, plus 8.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 3passed8h 44m into the runtook 14m 3s
startup 262.6 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 262.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#263746\" stroke-width=\"4\" stroke-linejoin=\"round\" stroke-linecap=\"round\">\n <!--Left person's legs: two dark blue rectangles forming the lower limbs, positione ...[clipped]
- ✓valid SVG · 4,432 of 32,768 bytes, 29 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 14.8 s, about 235 s for 254, plus 8.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted the BF16 checkpoint in /workspace/output/model/. Final self-check passed.