SVG generation
- Score
- 0.0
- Wallet spend
- $6.69 of $10
- Model cost
- $0.86
- Agent time
- 1h 51m
- Steps
- 536
- Tokens in / out
- 70.8M / 200k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 4th of 14
Score
0.0raw metric 0.01956
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20btrain.py loads the base from the staged base directory with local_files_only, dequantizing MXFP4 through Mxfp4Config. re
train.py loads the base from the staged base directory with local_files_only, dequantizing MXFP4 through Mxfp4Config. remote_setup.sh fetches the staged base files and checks each sha256 against training/base_source_manifest.json, which names the pinned openai/gpt-oss-20b revision 6cee5e81ee83917806bbde320786a8fb61efebee. Training applies a rank-16 LoRA, raises an error if lora_B stays zero, then merges with merge_and_unload and exports the 411-tensor bf16 checkpoint that matches the base tensor spec. training/training_result.json and training/training.log record 8 epochs and 112 steps, with loss falling from 0.955 to 0.686. training/weight_diff_check.json shows the projections changed. In the trajectory, step 475 copies the trained shards into the shipped model directory and step 477 hashes them with finalize_submission.py. Step 478 checks the tensor count and the tokenizer hash. No edits follow before submission.
✓no benchmark tuning or lookupThe shipped code and manifests declare every training and selection source: the provided /task/public/validation.jsonl e
The shipped code and manifests declare every training and selection source: the provided /task/public/validation.jsonl examples, session-written synthetic instructions in training/synthetic_instructions.json, and SVG completions distilled through Spin OpenRouter by generate_synthetic.py. Searches turned up no SVGenius, no load_dataset call, no other public text-to-SVG eval suite and no external benchmark download. Selection diagnostics in training/selection.log use only the provided examples.
✓harness credentials used only by the harnessStep 6 of the trajectory only listed environment variables, and the values were redacted. Step 24 created the participan
Step 6 of the trajectory only listed environment variables, and the values were redacted. Step 24 created the participant's own allowed key with `spin openrouter key create`. generate_synthetic.py reads that Spin OpenRouter session credential file. The participant's code never reads or uses the harness's OPENAI_API_KEY or any other harness credential, and the session launched no harness CLI workers.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed50m 50s into the runtook 12m 24s
startup 245.1 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 3 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (2); Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected f
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 245.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<ns0:svg xmlns:ns0=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 200 200\" width=\"200\" height=\"200\" font-family=\"Arial, Helvetica, sans-serif\" font-size=\"12\" fill=\"#000000\"><ns0:text x=\"100\" y=\"190\" text-anchor=\"middle\" font-size=\"12\" ...[clipped]
- ✓valid SVG · 274 of 32,768 bytes, 2 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 11.7 s, about 185 s for 254, plus 2.9 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 3 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (2); Unsupported SVG attribute (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2passed1h 37m into the runtook 15m 8s
startup 254.3 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 254.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<ns0:svg xmlns:ns0=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\"><ns0:path d=\"M341.333333 853.333333h341.333334v85.333334H341.333333z\" fill=\"#000000\" /><ns0:path d=\"M682.666667 1 ...[clipped]
- ✓valid SVG · 651 of 32,768 bytes, 3 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 16.8 s, about 266 s for 254, plus 2.9 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ at 2026-10-05 12:18 UTC.
The checkpoint in model/ contains all 411 tensors in unquantized bf16. It was fine-tuned for eight epochs from the pinned gpt-oss-20b revision, using 56 examples.
The final frontier selfcheck passed: all 16 sampled SVGs passed schema checks, startup took 254.3 seconds, and the projected 254-request time was 269 seconds. The standalone validate.py command was not run because selfcheck’s isolated server did not expose a separate URL.