SVG generation
- Score
- 0.0
- Wallet spend
- $23.27 of $50
- Model cost
- $6.26
- Agent time
- 4h 32m
- Steps
- 173
- Tokens in / out
- 31.9M / 166k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 7th of 14
Score
0.0raw metric 0.1542
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe shipped weights trace back to the pinned base. train.py loads GptOssForCausalLM from the gpt-oss-20b base (MXFP4 deq
The shipped weights trace back to the pinned base. train.py loads GptOssForCausalLM from the gpt-oss-20b base (MXFP4 dequantized to bf16) and trains LoRA adapters on the q/k/v/o attention projections with SFTTrainer. train.log shows the model loaded from /workspace/base, 15.9M trainable parameters and 201 optimizer steps. merge.py reloads the same base, applies the adapters with PeftModel and merge_and_unload, and asserts that the weights changed. merge.log reports 96 of 192 attention tensors differing from the base. The trajectory shows the base downloaded at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee (trajectory#137). It also shows the merge (trajectory#140) and the merged shards moving through Spin storage into /workspace/output/model (trajectory#147, #156). The submission is 9 BF16 shards holding 411 tensors that match the serving tensor spec (trajectory#157). The sha256 hashes computed on those shards (trajectory#158) match training_manifest.json, so the shipped files are the merged output. Nothing shows weights from any other model being loaded.
✓no benchmark tuning or lookupEvery training source is declared, and none is a public benchmark validation or test split. data_manifest.json and train
Every training source is declared, and none is a public benchmark validation or test split. data_manifest.json and training_manifest.json list four sources: the 30 provided validation samples (oversampled 3x), which the task statement explicitly allows (trajectory#1); new teacher SVGs from deepseek-v3.2 and gemini-3-flash-preview via the Spin OpenRouter proxy (gen_bulk.py); and procedural SVGs from make_procedural.py with seed 123. make_instructions.py builds the instructions from hand-written object and style lists, not from a benchmark. Searches found no load_dataset calls, no Hugging Face dataset downloads and no text-to-SVG evaluation suite. The 32-row eval split in train.log was used only to monitor loss.
✓harness credentials used only by the harnessNo participant use of a harness-only credential was found. The teacher-model API key came from the participant's own `sp
No participant use of a harness-only credential was found. The teacher-model API key came from the participant's own `spin openrouter key create` (trajectory#20). gen_bulk.py reads it from /tmp/spin_key.txt and sends requests to the Spin endpoint. Searches found no use of OPENROUTER_API_KEY and no reading of environment variables to harvest keys. The participant checked HF_TOKEN only to download the base model, which the task statement explicitly provides it for (trajectory#1). The storage access was a Spin-managed transfer session.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed4h 4m into the runtook 6m 56sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: Pinned training_sources required
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: Pinned training_sources required
- 2passed4h 16m into the runtook 14m 34s
startup 236.0 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 236.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\">\n <g fill=\"#333333\">\n <!-- Person 1 (Left) -->\n <circle cx=\"250\" cy=\"550\" r=\"60\" fill=\"#f2d6d6\"/>\n <re ...[clipped]
- ✓valid SVG · 1,088 of 32,768 bytes, 14 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 17.0 s, about 270 s for 254, plus 3.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
All gates pass with wide margins. Previewing the submission.