SVG generation
- Score
- 0.0
- Wallet spend
- $84.47 of $200
- Model cost
- $40.99
- Agent time
- 9h 59m
- Steps
- 373
- Tokens in / out
- 70.4M / 162k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 6th of 14
Score
0.0raw metric 0.1787
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bThe submission was fine-tuned from openai/gpt-oss-20b. - **Base download:** The trajectory shows the base being download
The submission was fine-tuned from openai/gpt-oss-20b. - **Base download:** The trajectory shows the base being downloaded from huggingface.co at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee. - **Loading and training:** convert_base.py and train_sft.py load the MXFP4 base, dequantize it to bf16 and check it against the 411-tensor GptOssForCausalLM layout. They then train the attention projections, sinks, layernorms and router, and save a full 411-tensor checkpoint. - **Weights actually moved:** After run 3, the trajectory (trajectory#244) shows non-zero changes from the base: q_proj about 3.9e-4 and router about 3.7e-4. Layernorms were unchanged (0.0). - **Lost and rebuilt shard:** The compute lease expired and one shard was lost (trajectory#316). Shard 8 holds the layer-23 experts, lm_head and 3 layernorms. It was rebuilt from the dequantized base (trajectory#317, trajectory#323). This is legitimate because the experts and lm_head were frozen and the layernorms had zero change. The rebuild is openly declared in architecture.json and train_summary.json. - **Packaging:** The packaged model was copied from ckpt-run3 (trajectory#329). training_manifest.json records sha256 hashes for every shard, and the hash check passed (trajectory#370). - **Minor inconsistency:** The shipped train_sft.py uses a plain AdamW step on bf16 parameters. The fp32 master-weight fix described in the session does not appear in it. The recorded changes from the base still show real fine-tuning, so this does not establish a violation.
✓no benchmark tuning or lookupThere is no sign of benchmark tuning or lookup. - **No benchmark access:** Neither the code nor the trajectory mentions
There is no sign of benchmark tuning or lookup. - **No benchmark access:** Neither the code nor the trajectory mentions SVGenius or pulls any benchmark dataset (no load_dataset, snapshot_download or hf download calls for it). - **Training data source:** The data is synthetic. - Instructions were generated by claude-haiku-5.5 (gen_instructions.py). - SVG answers were generated by claude-sonnet-5.5 through the participant's spin openrouter allowance, then validated and canonicalized (gen_svgs.py, prep_data.py). - The provided public examples were also used, and they are allowed. - **Declared provenance:** These sources are listed in architecture.json and training_manifest.json, including the hash of train2.jsonl. - **Few-shot examples:** These come only from the supplied public validation set.
✓harness credentials used only by the harnessThe participant did not use the harness credentials. - **No harness key in code:** None of the shipped code reads OPENRO
The participant did not use the harness credentials. - **No harness key in code:** None of the shipped code reads OPENROUTER_API_KEY. The only _BASE_URL match is help text in evaluate.py. - **Participant's own key:** The key used for teacher and judge calls came from `spin openrouter key create` (trajectory#17). This is the participant's own Spin allowance, not the harness credential. It was passed as SPIN_OR_KEY (trajectory#247, trajectory#367). - **No harness tools:** The participant did not launch any harness CLI or harness-only endpoint.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed8h 48m into the runtook 11m 42s
startup 165.7 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 165.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\">\n <defs>\n <linearGradient id=\"b\" x1=\"0\" y1=\"0\" x2=\"0\" y2=\"1\">\n <stop offset=\"0\" stop-color=\"#f7c9a0\" />\n <stop offset=\"1\" stop-color=\"#eaa87a\" />\n ...[clipped]
- ✓valid SVG · 1,755 of 32,768 bytes, 27 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 10.7 s, about 170 s for 254, plus 4.1 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Everything verifies: hashes OK, tokenizer byte-identical to base, no serve.sh, all evidence files present, selfcheck passed. Run 4 is unrecoverable (dead box), run 5 is worse — run 3 is the final choice. Let me hand it in.