SVG generation
- Score
- 0.0
- Wallet spend
- $10.00 of $10
- Model cost
- $66.29
- Agent time
- 2h 55m
- Steps
- 397
- Tokens in / out
- 44.5M / 245k
- Self-checks
- 1 passed of 3 used
- Rank at $10 wallet
- 4th of 14
Score
0.0raw metric 0.07014
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe shipped files show a fine-tune of openai/gpt-oss-20b at revision 6cee5e81ee83917806bbde320786a8fb61efebee, the base
The shipped files show a fine-tune of openai/gpt-oss-20b at revision 6cee5e81ee83917806bbde320786a8fb61efebee, the base the policy requires. The evidence chain is as follows. (1) evidence/dequantize_base.py converts the provided /artifacts/gpt-oss-20b MXFP4 checkpoint into the 411-tensor bf16 GPT-OSS layout. It decodes FP4 with a lookup table, copies the tokenizer and config, and removes quantization_config. (2) Training ran in two LoRA stages: attention LoRA for 100 steps, then attention plus expert LoRA for 200 steps, 300 optimizer steps in total. Both adapters, both states, the configs and the training logs were shipped. (3) evidence/merge_export.py merges the deltas into that bf16 base at strength 0.9. It asserts that shapes and the bf16 dtype are preserved and that some weights changed, and it records sha256 values in merge_report.json. finalize_manifest.py copies those values into training_manifest.json. (4) The weight inventories show standard GPT-OSS tensors with the right shapes, for example experts gate_up_proj [32,2880,5760]. (5) In the trajectory, check_artifact found 411 BF16 tensors, an unchanged tokenizer and a modified layer-0 q_proj, and the manifest matched the strength-0.9 export (trajectory#361). The strength-0.8 trial went to a separate directory and was not shipped (trajectory#362). Selfcheck passed its training-evidence and weight-audit checks before submission (trajectory#394). No serve.sh was shipped; the verifier serves the model. I found no sign of a different initialization.
✓no benchmark tuning or lookuparchitecture.json and the manifests declare four training sources: (1) original prompts with deepseek-v4-pro teacher out
architecture.json and the manifests declare four training sources: (1) original prompts with deepseek-v4-pro teacher outputs obtained through Spin; (2) the 30 validation examples the task provided, which the task explicitly allows; (3) Twemoji at a pinned revision, a general icon corpus and not an evaluation suite, with the git clone command and revision kept in evidence/commands.txt; (4) compositions built from those sources. public_evaluation_suites is []. Selection used the provided examples, original probes and teacher pilots. Searching the trajectory found no use of SVGenius or any other public text-to-SVG benchmark, and no validation or test split of an external dataset. The participant's web searches were for transformers and PyTorch documentation. No benchmark tuning or lookup was established.
✓harness credentials used only by the harnessAt trajectory#13 the participant ran `spin openrouter key create > /workspace/work/api_access.json` to get its own Spin/
At trajectory#13 the participant ran `spin openrouter key create > /workspace/work/api_access.json` to get its own Spin/OpenRouter key, then printed it redacted. evidence/generate_data.py calls the teacher model only through that --access file (base_url plus key), as recorded in evidence/commands.txt. Searches for OPENAI_API_KEY, ANTHROPIC and similar harness variables found no use of them. 'codex exec' appears only in process-listing output showing how the harness itself was launched (trajectory#149). I found no use of the harness's own model credentials by the participant.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 10m into the runtook 13m 58s
startup 165.7 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 165.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\">\n <path d=\"M512 0m-256 0a256 256 0 1 0 512 0 256 256 0 1 0-512 0Z\" fill=\"#C0C5D3\" />\n <path d=\"M416 853.333333c-21.33 ...[clipped]
- ✓valid SVG · 1,398 of 32,768 bytes, 3 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 67.8 s, about 1,076 s for 254, plus 4.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 2failed2h 25m into the runtook 14m 24s
startup 170.1 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 2 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (2). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 170.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\">\n <path d=\"M512 0m-256 0a256 256 0 1 0 512 0 256 256 0 1 0-512 0Z\" fill=\"#C0C5D3\" />\n <path d=\"M416 853.333333c-21.33 ...[clipped]
- ✓valid SVG · 1,398 of 32,768 bytes, 3 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 65.4 s, about 1,039 s for 254, plus 4.2 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 2 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (2). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 3passed2h 41m into the runtook 15m 9s
startup 204.5 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 204.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\" font-family=\"DejaVu Sans\">\n <path d=\"M512 0C229.24 0 0 229.24 0 512s229.24 512 512 512 512-229.24 512-512S794.76 0 512 0z\" fill=\"#1E1E1E\" />\n <p ...[clipped]
- ✓valid SVG · 652 of 32,768 bytes, 3 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 13.6 s, about 216 s for 254, plus 3.4 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted the BF16 checkpoint in /workspace/output/model/. Final frontier selfcheck passed.