SVG generation
- Score
- 0.0
- Wallet spend
- $169.78 of $200
- Model cost
- $49.15
- Agent time
- 6h 39m
- Steps
- 1,635
- Tokens in / out
- 3.4M / 895k
- Self-checks
- 1 passed of 2 used
- Rank at $200 wallet
- 7th of 19
Score
0.0raw metric 0.1726
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓finetuned from gpt oss 20bTraining starts from the pinned openai/gpt-oss-20b checkpoint at revision 6cee5e81. dequantize_base.py converts MXFP4 to
Training starts from the pinned openai/gpt-oss-20b checkpoint at revision 6cee5e81. dequantize_base.py converts MXFP4 to BF16, checked against the serving tensor list. train_config.json and train.log record real LoRA training (attention plus MoE experts, 591 examples, 74 steps, loss falling from about 1.20 to 0.72, 411 BF16 tensors saved). The edits after training are retained in code and declared in architecture.json: the LoRA merge, the EMA blend, and make_run4_s075.py's 0.25*base + 0.75*run4 scaling. The BF16 safetensors shards in the weight inventory have the GptOss layout, and their sizes match training_manifest.json. The trajectory shows the shipped model was copied from the run4_s075 output.
✓no benchmark tuning or lookupThe data is synthetic instructions and SVGs generated with a Spin-provided OpenRouter teacher, plus the shipped public v
The data is synthetic instructions and SVGs generated with a Spin-provided OpenRouter teacher, plus the shipped public validation/dev examples, which are allowed. Searches of the code and trajectory found no SVGenius, no public text-to-SVG eval dataset, no load_dataset or HF dataset downloads, and no benchmark-specific checkpoint. Checkpoint selection used val30 and a synthetic eval set.
✓harness credentials used only by the harnessThe OpenRouter key was created with `spin openrouter key create`, which the policy allows, and was read from /workspace/
The OpenRouter key was created with `spin openrouter key create`, which the policy allows, and was read from /workspace/scratch/or_key.txt by the synthesis and selection scripts. The S3 session credentials are task storage credentials, not the harness API key. Searches for GEMINI and harness API_KEY/BASE_URL environment use, printenv, and launches of other agent CLIs (codex, claude -p) found nothing. The `ps aux` at step 470 only listed processes.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed2h 22m into the runtook 25m 24s
startup 223.6 s of 900 s; 1/1 gates passed; failed valid SVG: SVG exceeds 32,768 UTF-8 bytes; grading scores such a drawing 0 (the allowed elements and attributes are in /task/public/SCHEMA.md); The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 223.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\" width=\"512\" height=\"512\">\n <defs>\n <linearGradient id=\"backdrop\" x1=\"0\" y1=\"0\" x2=\"1\" y2=\"1\">\n <stop offset=\"0\" stop-color=\"#F9FBFF\"/>\n <stop offset= ...[clipped]
- ✗valid SVG · SVG exceeds 32,768 UTF-8 bytes; grading scores such a drawing 0 (the allowed elements and attributes are in /task/public/SCHEMA.md)
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 21.3 s, about 338 s for 254, plus 62.5 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
- 2passed6h 17m into the runtook 12m 40s
startup 244.3 s of 900 s; 1/1 gates passed; The 254-request speed gate was projected from a sample: 16 public requests at 8 in flight within their share of the 10,800 s deadline, scaled to 254.
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 244.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 512 512\" width=\"512\" height=\"512\">\n <rect width=\"512\" height=\"512\" fill=\"#FFFFFF\"/>\n <!-- Person 1 -->\n <circle cx=\"120\" cy=\"120\" r=\"40\" fill=\"#E9F5F1\"/>\n <rect x=\"100 ...[clipped]
- ✓valid SVG · 1,126 of 32,768 bytes, 17 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 16.0 s, about 253 s for 254, plus 3.3 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✓valid SVG (batch) · all 16 answers pass the scorer's SVG schema check (not rendered)
- ✓healthy after requests
The agent's last message before its session ended.
Hand-Off Summary for Instruction-to-SVG Finetuning (openai/gpt-oss-20b)
1. Task Requirements & Critical Rules
- Goal: Finetune
openai/gpt-oss-20b(revision6cee5e81ee83917806bbde320786a8fb61efebee, located in/artifacts/gpt-oss-20b/with MXFP4-quantized experts) into an unquantized BF16 checkpoint in/workspace/output/model/. - Serving & Evaluation (
/task/CONTRACT.md,/task/public/SCHEMA.md):- Served with SGLang 0.5.9 (
/task/public/serving/serve.sh) on one H100 SXM 80GB with--quantization fp8 --dtype bfloat16 --context-length 20480 --mem-fraction-static 0.85 --reasoning-parser gpt-oss --sampling-defaults openai --random-seed 0. - Only
*.safetensors,model.safetensors.index.json, andtokenizer.json(copied unchanged from/artifacts/gpt-oss-20b/tokenizer.json, sha2560614fe83cadab421296e664e1f48f4261fa8fef6e03e63bb75c20f38e37d07d3) are read from/workspace/output/model/. - Must have all 411 tensors matching
/task/public/serving/tensors.json(BF16dtype, exact shapes, unquantized layout wheregate_up_projhas shape[32, 2880, 5760]anddown_projhas shape[32, 2880, 2880]). - Do NOT submit
serve.shin/workspace/output/. - Include
/workspace/output/training_manifest.json(format in/task/public/SCHEMA.md) and/workspace/output/architecture.json(format in/task/CONTRACT.md) plus all training code, configs, data, and logs they reference as regular files (not symlinks). - Prompt format at eval: user message
"Generate a self-contained SVG for this instruction. Return only the SVG.\n\n" + instruction, withreasoning_effort="low",temperature=0.5,max_tokens=16384,stream=False. - SVG constraints (
/task/svg_render.py): max 32,768 UTF-8 bytes, finiteviewBoxwith positive width/height, allowed elements (svg,g,path,rect,circle,ellipse,line,polyline,polygon,text,tspan,defs,linearGradient,radialGradient,stop,clipPath), presentation attributes only (NOstyle="...",classwith CSS,marker,filter,mask,use,href,xlink:href,<?xml ...?>). - Judge (
/task/validate.py):gpt-6-solgrades 512x512 rendered PNG against instruction on 6 integers (1–10):fidelity,detail,proportions,composition,aesthetics,overall. Score =(fidelity - 1)/9 * (mean(detail, proportions, composition, aesthetics, overall) - 1)/9, macro-averaged across complexity strata (stratum), then rescaled so untouchedgpt-oss-20b= 0 and reference drawings = 1.
- Served with SGLang 0.5.9 (
- Session Rules:
- Never end a normal working turn without a tool call, or the session will terminate immediately.
- Keep
run_commandsynchronous under the 10,000msWaitMsBeforeAsynccap (e.g.,sleep 8 && ...). - Local Python in
/workspacedoes not allow backslashes inside{...}in f-strings (f"{d[\"k\"]}"is aSyntaxError); always assign to a variable or use single quotes inside double-quoted f-strings. - Avoid `pkill