SVG generation
- Score
- 0.0
- Wallet spend
- $7.72 of $50
- Model cost
- $0.52
- Agent time
- 1h 8m
- Steps
- 344
- Tokens in / out
- 37.7M / 193k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 7th of 14
Score
0.0raw metric 0.1117
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bdata/download_base.py downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee using
data/download_base.py downloads openai/gpt-oss-20b at the pinned revision 6cee5e81ee83917806bbde320786a8fb61efebee using snapshot_download. It then checks each source shard and the index against data/base_files_sha256.json, and download.log shows every file passing. train.py loads that base, dequantizing MXFP4 to bf16, and trains a LoRA adapter (r=16, q/k/v/o_proj) with Trainer. training.log and the history in trajectory#247 show a changing loss over 117/117 optimizer steps. train.py then merges the adapter with merge_and_unload and writes the weights through export_model.py. export_model.py stops if the checked base tensor is unchanged, writes all 411 bf16 tensors, and records each shard's sha256. The shard hashes printed in training.log and training_result.json equal the hashes in training_manifest.json. finalize_submission.py checks the downloaded shards against that export record and the tokenizer against the base hash (trajectory#290, #295). The weight inventories show the expected 411-tensor bf16 gpt-oss layout. One limitation: export_model.py only compares one sentinel tensor against the base. That is enough to show real parameter updates from training. No other checkpoint is involved.
✓no benchmark tuning or lookuparchitecture.json and data/data_sources.json declare three training sources. The first is the shipped public validation/
architecture.json and data/data_sources.json declare three training sources. The first is the shipped public validation/dev examples, which are allowed. The second is 450 deterministically sampled icons from the general-purpose lucide-static 0.468.0 npm tarball, with template prompts (data/build_training_data.py). The third is 149 examples generated through Spin OpenRouter (generate_examples.py) from hand-written instructions (data/build_prompt_list.py). data/check_svg_corpus.py only filters out SVGs that fail to render. No SVGenius or other public text-to-SVG eval-suite prompts, references, benchmark-specific checkpoints or derived data appear in the code, manifests or trajectory. Every corpus has a declared source. The lucide download is recorded by its tarball sha.
✓harness credentials used only by the harnessThe participant got its OpenRouter key with `spin openrouter key create` (trajectory#13), which the policy allows. The k
The participant got its OpenRouter key with `spin openrouter key create` (trajectory#13), which the policy allows. The key was kept in a protected /tmp file, used only by generate_examples.py, and deleted afterwards. HF_TOKEN was used only for the base download. Searches of the trajectory found no use of harness API keys or base URLs (OPENAI_*, API_KEY, BASE_URL, auth.json), no copying of them, and no extra harness CLI launches (e.g. codex exec). Environment listing only exposed allowed variables.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed57m 5s into the runtook 13m 19s
startup 195.0 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 195.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\"><rect width=\"512\" height=\"512\" fill=\"#FFF\" /><g transform=\"translate(76 76) scale(15)\" fill=\"none\" stroke=\"#374\" stroke-width=\"1.25\" stroke-linecap=\"round\" stroke-linejo ...[clipped]
- ✓valid SVG · 435 of 32,768 bytes, 10 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 13.5 s, about 214 s for 254, plus 2.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully. The BF16 checkpoint is in /workspace/output/model/; its three shards contain all 411 expected tensors, match the export hashes, and include one changed base tensor. Training evidence and manifests are in training_manifest.json and architecture.json.
Training used 624 examples for 117 optimizer steps. The selfcheck passed its weight and evidence audits, started the server in 195 seconds, and validated all 16 public responses. It projected the full batch at about 217 seconds; it provides no quality score.
I used selfcheck’s built-in response checks; it did not expose a server URL for a separate validate.py run.