SVG generation
- Score
- 0.0
- Wallet spend
- $48.56 of $50
- Model cost
- $22.20
- Agent time
- 11h 10m
- Steps
- 453
- Tokens in / out
- 44.3M / 246k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 8th of 19
Score
0.0raw metric 0.1678
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20barchitecture.json names the base as openai/gpt-oss-20b at revision 6cee5e81ee83917806bbde320786a8fb61efebee. training/ba
architecture.json names the base as openai/gpt-oss-20b at revision 6cee5e81ee83917806bbde320786a8fb61efebee. training/base_download.json records a pinned snapshot_download of that revision, and training/dequantize_base.py converts its MXFP4 weights to bf16 at /workspace/dequant_base. training/train_svg.py is a real PEFT LoRA training loop (r=32 on attention and lm_head, plus custom expert LoRA on the MoE gate_up/down projections). training/train_expanded.sh runs it on combined_teacher.jsonl, and training/train_expanded.log shows the run starting from /workspace/dequant_base with the loss falling. The selected adapter was built by training/average_adapters.py as 0.75 × epoch4 + 0.25 × epoch5. Both are this session's own adapters, so the average is an allowed edit. training/merge_plus_epoch45w75.log merges that adapter into dequant_base. checkpoint_audit.json lists 411 BF16 tensors with shard sha256 values that match training_manifest.json, confirms the tokenizer matches the base, and shows the q_proj weights differ from the base, which matches the merge log. In the session, the merged plus_epoch45w75 model was moved into /workspace/output/model at trajectory#366 and submitted unchanged at trajectory#453. No other initialization was found.
✓no benchmark tuning or lookuptraining/combined_manifest.json and training/data_manifest.json declare three training sources: main_teacher (369 exampl
training/combined_manifest.json and training/data_manifest.json declare three training sources: main_teacher (369 examples), extra_teacher (418 generated, with 24 held back for development) and the 30 task-provided public validation examples, which the task permits. Instructions came from handwritten taxonomies in training/make_instructions.py. The teacher prompt in training/teacher_instructions.py forbids copying from any dataset or benchmark. Searches of the code and the session found no benchmark datasets, no load_dataset calls, no SVGenius references, no Hugging Face dataset downloads and no git clones. Evaluation used only the public validation set. The gpt-6-sol teacher generation through the Spin OpenRouter key is declared, but it is verified only through the retained scripts and manifests, not through independent download logs.
✓harness credentials used only by the harnessTeacher and judge calls (training/teacher_generate.py, training/teacher_instructions.py, training/judge_gateway_adapter.
Teacher and judge calls (training/teacher_generate.py, training/teacher_instructions.py, training/judge_gateway_adapter.py) read /workspace/openrouter_key.json. That key came from the participant's own `spin openrouter key create` in the session (trajectory chunk steps-8-20), and that budget is allowed. The evaluation scripts set MLE_JUDGE_API_KEY=diagnostic-loopback-only rather than any harness key. Searches of the session found no use of ANTHROPIC or OPENAI_API_KEY harness credentials, no environment dumps and no extra `claude -p` harness sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed1h 30m into the runtook 11m 18s
startup 150.5 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 150.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <defs>\n <linearGradient id=\"skin\" x1=\"0\" y1=\"0\" x2=\"1\" y2=\"1\">\n <stop offset=\"0\" stop-color=\"#f7d5a8\"/>\n <stop offset=\"1\" stop-color=\"#e9a96b\"/>\n ...[clipped]
- ✓valid SVG · 9,986 of 32,768 bytes, 77 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 73.2 s, about 1,162 s for 254, plus 16.4 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2passed7h 54m into the runtook 12m 19s
startup 143.9 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 143.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <g stroke=\"#34414b\" stroke-linecap=\"round\" stroke-linejoin=\"round\">\n <!-- Person on the left -->\n <path d=\"M68 273 C68 295 53 309 51 331 L45 391 Q48 414 70 420 L106 4 ...[clipped]
- ✓valid SVG · 5,054 of 32,768 bytes, 32 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 15.6 s, about 248 s for 254, plus 9.5 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
The selected soup has passed selfcheck, and the corrected recovery archive now verifies end to end with both provenance manifests and all candidate reports. I’m submitting the unchanged /workspace/output directory.