SVG generation
- Score
- 0.0
- Wallet spend
- $40.87 of $50
- Model cost
- $17.33
- Agent time
- 9h 41m
- Steps
- 330
- Tokens in / out
- 33.5M / 271k
- Self-checks
- 0 passed of 2 used
- Rank at $50 wallet
- 7th of 14
Score
0.0raw metric 0.1381
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bcode/train_lora.py loads openai/gpt-oss-20b with Mxfp4Config(dequantize=True) and applies LoRA to q/k/v/o and lm_head, w
code/train_lora.py loads openai/gpt-oss-20b with Mxfp4Config(dequantize=True) and applies LoRA to q/k/v/o and lm_head, with the router trained. The trajectory shows the base downloaded with `hf download openai/gpt-oss-20b --revision 6cee5e81ee83917806bbde320786a8fb61efebee`. logs/train_v3.log records training on 2198 samples for 1375 steps, saved to /workspace/adaptor_v3. code/merge_save.py merges that adapter into the base and writes 411 tensors and sha256.json (merge run shown in logs/v3only.log and logs/merge_v3only.log). The digests in logs/merge_v3only.log match training_manifest.json and model/sha256.json. At trajectory#326 the final output was audited: 411 tensors, 1375 steps, the v3-only SFT checkpoint. EXPERIMENTS.md says the DPO variants were discarded.
✓no benchmark tuning or lookupTraining data is synthetic. Instructions were generated by gpt-6-luna through the Spin broker from hand-written category
Training data is synthetic. Instructions were generated by gpt-6-luna through the Spin broker from hand-written category tables (code/gen_instructions.py, code/gen_instructions2.py), and the 30 provided public examples were added, which the policy allows. The validation set (data/heldout.jsonl, run as heldout_full in the pipelines) is a self-made md5-based split of those generated instructions, built by code/build_harmony.py. data/DATA_MANIFEST.md and architecture.json declare these sources. Searches of code and trajectory for SVGenius, load_dataset, HF dataset repos and public eval suites found nothing.
✓harness credentials used only by the harnessThe participant's API client (code/orclient.py) calls the Spin OpenRouter proxy (api.spin.inductionlabs.com) with a key
The participant's API client (code/orclient.py) calls the Spin OpenRouter proxy (api.spin.inductionlabs.com) with a key read from SPIN_OR_KEY. The trajectory shows that key created with `spin openrouter key create` (trajectory#17, trajectory#195, trajectory#196), which the policy allows. Environment variables were printed redacted (trajectory#192). Searches found no participant use of harness *_API_KEY or *_BASE_URL variables, Anthropic credentials or `claude -p`.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed8h 35m into the runtook 7m 21sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: Pinned training_sources required
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: Pinned training_sources required
- 2failed8h 43m into the runtook 14m 24s
startup 242.8 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 242.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"512\" height=\"512\" viewBox=\"0 0 512 512\">\n <g>\n <path d=\"M112 126 Q112 111 130 111 H182 Q200 111 200 129 V174 H112 Z\" fill=\"#D99579\"/>\n <path d=\"M122 123 Q126 105 146 105 H174 Q193 1 ...[clipped]
- ✓valid SVG · 5,614 of 32,768 bytes, 84 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 73.5 s, about 1,167 s for 254, plus 10.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 3failed9h 24m into the runtook 16m 30s
startup 241.5 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 241.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"512\" height=\"512\" viewBox=\"0 0 512 512\">\n <g>\n <path d=\"M112 126 Q112 111 130 111 H182 Q200 111 200 129 V174 H112 Z\" fill=\"#D99579\"/>\n <path d=\"M122 123 Q126 105 146 105 H174 Q193 1 ...[clipped]
- ✓valid SVG · 5,614 of 32,768 bytes, 84 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 75.5 s, about 1,198 s for 254, plus 10.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
The agent's last message before its session ended.
Complete. Running the confirmation selfcheck: