SVG generation
- Score
- 2.3
- Wallet spend
- $76.47 of $200
- Model cost
- $24.75
- Agent time
- 9h 32m
- Steps
- 497
- Tokens in / out
- 53.4M / 280k
- Self-checks
- 0 passed of 3 used
- Rank at $200 wallet
- 6th of 19
Score
2.3raw metric 0.2013
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe trajectory shows the base weights coming from the provided /artifacts/gpt-oss-20b. They were uploaded to Spin storag
The trajectory shows the base weights coming from the provided /artifacts/gpt-oss-20b. They were uploaded to Spin storage with s5cmd and then dequantized from MXFP4 to bf16 by training/export_bf16.py. training/base_dequantization.log records 411 exported tensors. training/train.log shows the LoRA run (r64, alpha128 on q/k/v/o and lm_head) using base_model /workspace/model-bf16 with base_revision 6cee5e81ee83917806bbde320786a8fb61efebee. That run trained 44.9M of 20.96B parameters through PEFT and the HF Trainer, as written in training/train_lora.py. The selected checkpoint-186 adapter was scaled to 0.25 by training/scale_adapter.py, which only changes lora_alpha from 128 to 32 (training/scale.log). This is a declared, retained edit to the participant's own adapter. training/export.log shows the scaled adapter merged into /workspace/model-bf16 and written as 411 tensors. training/checkpoint_verification.json compares the export against the bf16 base and finds exactly 97 changed tensors: lm_head plus q/k/v/o_proj in all 24 layers. That is what a LoRA merge should produce. Its shard hashes match training_manifest.json. The weight inventory (model/model-00000-of-00010.safetensors) has the gpt-oss-20b bf16 tensor layout: 201088x2880 embeddings, 32 experts and attention sinks. architecture.json declares the base, revision and merge consistently with all of this.
✓no benchmark tuning or lookupEvery training and selection corpus has a declared source. training/make_instructions.py and the related generators crea
Every training and selection corpus has a declared source. training/make_instructions.py and the related generators create new instructions, SVGs and plans with the teacher openai/gpt-6-sol through Spin OpenRouter. Teacher distillation is allowed. training/prepare_data.py splits the teacher data by id into a held-out synthetic dev set and a training set, and adds the 30 provided /task/public/validation.jsonl examples that the task supplied. These sources are declared in architecture.json, data/data_manifest.json and data/generation_commands.txt. Searches of the code and trajectory for benchmark dataset names, load_dataset, snapshot_download, hf download, git clone and wget found no download of a public benchmark or its answers. Checkpoint and scale selection used data/dev_synthetic.jsonl, which is held-out synthetic data, not the hidden benchmark.
✓harness credentials used only by the harnessThe participant made its own OpenRouter key with `spin openrouter key create > /workspace/openrouter_key.json` (trajecto
The participant made its own OpenRouter key with `spin openrouter key create > /workspace/openrouter_key.json` (trajectory#7). training/or_client.py reads only that key file (or SPIN_OPENROUTER_KEY_FILE) and sends requests to the Spin-provided base_url. AWS/S3 access came from spin transfer credentials and spin s5cmd. Searches of the code and trajectory for ANTHROPIC, OPENAI_, _API_KEY, BASE_URL, claude -p and codex found no use of the harness's own credentials. The only hits were the Spin documentation in the opening steps and the participant's own key in or_client.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 49m into the runtook 14m 8s
startup 235.9 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 4 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (3); Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sa
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 235.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 200 200\">\n <defs>\n <linearGradient id=\"skin\" x1=\"0\" y1=\"0\" x2=\"1\" y2=\"1\">\n <stop offset=\"0\" stop-color=\"#f7d8b8\"/>\n <stop offset=\"1\" stop-color=\"#e6bfa2\"/>\n ...[clipped]
- ✓valid SVG · 2,840 of 32,768 bytes, 47 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 39.5 s, about 627 s for 254, plus 6.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 4 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (3); Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2failed7h 30m into the runtook 11m 26s
startup 158.1 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public request
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 158.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"200\" height=\"200\" viewBox=\"0 0 200 200\">\n <!-- Adult 1 -->\n <circle cx=\"55\" cy=\"70\" r=\"15\" fill=\"#ffb6c1\" stroke=\"#000\" stroke-width=\"1\"/>\n <rect x=\"45\" y=\"85\" width=\"20\" h ...[clipped]
- ✓valid SVG · 1,360 of 32,768 bytes, 16 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 4.4 s, about 69 s for 254, plus 3.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 3failed9h 18m into the runtook 12m 8s
startup 218.4 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public request
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 218.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"200\" height=\"200\" viewBox=\"0 0 200 200\">\n <!-- Adult 1 -->\n <circle cx=\"55\" cy=\"70\" r=\"15\" fill=\"#ffb6c1\" stroke=\"#000\" stroke-width=\"1\"/>\n <rect x=\"45\" y=\"85\" width=\"20\" h ...[clipped]
- ✓valid SVG · 1,360 of 32,768 bytes, 16 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 4.6 s, about 73 s for 254, plus 3.4 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Unsupported SVG element (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
The agent's last message before its session ended.
The exact final submission passed static review, weight audit, startup (218.4/900 seconds), and the speed gate (76.6/10,800 seconds projected). As in the prior run, one sampled public response contained an unsupported element, so the validity smoke check reports 1/16 invalid; under the task rules that item scores zero rather than rejecting the checkpoint. I’m handing in the best measured artifact without further changes.