SVG generation
- Score
- 0.0
- Wallet spend
- $48.30 of $50
- Model cost
- $1.90
- Agent time
- 7h 58m
- Steps
- 860
- Tokens in / out
- 83.0M / 586k
- Self-checks
- 0 passed of 3 used
- Rank at $50 wallet
- 8th of 19
Score
0.0raw metric 0.126
The raw metric (judged drawing quality) is rescaled so that untouched gpt-oss-20b (0.188) scores 0 and the reference drawings (0.753) scores 100, clamped to that range.
Verifier details
- metric
- macro_graded_drawing_quality
- protocol
- svg-generation-v5-graded
- samples
- 254
- judge_model
- gpt-6-sol
- rubric_sha256
- 9cafaad1e10f5a288c5d2f84db71778fd31de1fd72dc6bb0e2415b81bd35c0af
- renderer_version
- rsvg-convert version 2.54.7
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓finetuned from gpt oss 20bThe weights trace back to the pinned base. - **Base:** code/train.py loads the local gpt-oss-20b base, which was downloa
The weights trace back to the pinned base. - **Base:** code/train.py loads the local gpt-oss-20b base, which was downloaded at the pinned revision 6cee5e81 (trajectory#45). MXFP4 is dequantized to bf16, LoRA (r64, alpha 128) is applied to q/k/v/o, and the run trains for 376 optimizer steps on train4.jsonl. The loss is logged in train.log. - **Merge:** code/merge_save.py merges adapter4 into the base weights with merge_and_unload, checks the result against the base tensor list, and writes 3 shards plus an index. It reuses the base tokenizer. The run is pipeline5.sh: adapter4 → merged4 (trajectory#559). - **Shipped files:** The sha256 of each shipped shard matches training_manifest.json and the hashes computed on the training box (cf2695cc…, ce358df8…, ee9d6df2…). The verify_submission output confirms the matches, with 411 tensors and a tokenizer identical to the base (trajectory#853). - **Serving:** No serve.sh was shipped; the verifier serves the model. Nothing suggests a different initialization.
✓no benchmark tuning or lookupNo benchmark data was used for training or lookup. - **Declared sources** (architecture.json): - instructions the sessio
No benchmark data was used for training or lookup. - **Declared sources** (architecture.json): - instructions the session wrote itself, by template expansion (code/gen_instructions.py) and with gpt-oss-120b (code/gen_model_instructions.py) - drawings from the grok-4.20 teacher (code/gen_svg.py), filtered by a judge model (code/judge.py) - the allowed public dev examples /task/public/validation.jsonl - **Logs:** Generation logs were kept for each source. - **Session search:** The trajectory shows no download of the SVGenius or benchmark test data, no HF datasets/load_dataset calls and no lookup of hidden answers. - **Evaluation:** The model was evaluated on a held-out split taken from the session's own pool (code/build_clean_eval.py).
✓harness credentials used only by the harnessThe session used only credentials it was allowed to use. - **Model calls:** Every model call (teacher, instruction write
The session used only credentials it was allowed to use. - **Model calls:** Every model call (teacher, instruction writer, judge, local scorer) goes through SPIN_OPENROUTER_URL/KEY. That key was minted in the session with `spin openrouter key create` and is allowed. - **Uploads:** Uploads to object storage used temporary credentials from a Spin transfer session.json (trajectory#113). - **Downloads:** HF_TOKEN was used only to download the model (trajectory#45). - **Failed call:** The curl at trajectory#708 sent a Spin key to a Spin endpoint and got a 401 rejection. It did not use a harness credential. - **Harness keys:** The session shows no use of the harness's ANTHROPIC/OPENAI keys and no extra claude/codex worker processes.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 10m into the runtook 6m 23sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json at the top of the submission
- 2failed1h 17m into the runtook 5m 56sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: [Errno 2] No such file or directory: '/tmp/frontier-selfcheck-5xpv9vi0/submission/training_manifest.json'
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: [Errno 2] No such file or directory: '/tmp/frontier-selfcheck-5xpv9vi0/submission/training_manifest.json'
- 3failed1h 34m into the runtook 5m 36sfree (stopped before any GPU time)
failed training evidence: Training evidence scan failed: Missing or escaping evidence path
2 gates
- ✓review prerequisites
- ✗training evidence · Training evidence scan failed: Missing or escaping evidence path
- 4failed1h 55m into the runtook 14m 50s
startup 228.6 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 228.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" viewBox=\"0 0 1024 1024\" width=\"512\" height=\"512\"><g><path fill=\"#FF6B6B\" d=\"M256 512C256 384 384 256 512 256s256 128 256 256c0 64-32 128-80 160l-88-32-88 32C288 640 256 576 256 512z\"/><path fill=\"#F ...[clipped]
- ✓valid SVG · 689 of 32,768 bytes, 8 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 67.6 s, about 1,073 s for 254, plus 2.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: SVG exceeds 32,768 UTF-8 bytes (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 5failed6h 16m into the runtook 11m 58s
startup 151.0 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 151.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"512\" height=\"512\" viewBox=\"0 0 1024 1024\"><g><path fill=\"#FF6B6B\" d=\"M256 512C256 384 352 288 480 288C608 288 704 384 704 512\" stroke=\"#1a1a1a\" stroke-width=\"48\" stroke-linejoin=\"round\"/ ...[clipped]
- ✓valid SVG · 959 of 32,768 bytes, 8 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 28.1 s, about 446 s for 254, plus 2.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
- 6failed7h 46m into the runtook 9m 11s
startup 107.0 s of 900 s; 1/1 gates passed; failed valid SVG (batch): 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md; The 254-request speed gate was projected from a sample: 16 public requests at 8
10 gates
- ✓review prerequisites
- ✓training evidence
- ✓weight audit
- ✓startup · GET /health answered 200 after 107.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0000", "svg": "<svg xmlns=\"http://www.w3.org/2000/svg\" width=\"512\" height=\"512\" viewBox=\"0 0 1024 1024\"><g><path fill=\"#FF6B6B\" d=\"M256 512C256 384 352 288 480 288C608 288 704 384 704 512\" stroke=\"#1a1a1a\" stroke-width=\"48\" stroke-linejoin=\"round\"/ ...[clipped]
- ✓valid SVG · 959 of 32,768 bytes, 8 elements, rendered by rsvg-convert version 2.54.7
- ✓254 requests, 8 in flight · projected from 16 of 254 requests at concurrency 8: 16 instructions drawn from the 30 public examples took 27.3 s, about 434 s for 254, plus 2.7 s for the first request. The public examples are simple icons only; detailed and complex held-out drawings usually need longer SVGs, so keep a wide margin
- ✓batch responses · all 16 answers parsed
- ✗valid SVG (batch) · 1 of 16 answers are not valid SVG and would score 0: Malformed SVG XML (1). The allowed elements and attributes are in /task/public/SCHEMA.md
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-21 03:36:12 UTC.
What was delivered
Weights (/workspace/output/model/): the round‑3 checkpoint — 3 bf16 safetensors shards holding exactly gpt‑oss‑20b's 411 tensors in the unquantized layout, model.safetensors.index.json, and a byte‑identical copy of the base tokenizer.json. The weights descend from openai/gpt-oss-20b@6cee5e81… only: embeddings, MoE experts and routers are bit‑identical to the base, and only the attention projections carry the LoRA deltas (logs/base_diff_report.json).
Training: LoRA r=64/α=128 on q/k/v/o_proj (31.9 M trainable params, 0.152%), 376 optimizer steps, 4 epochs, 1,502 rows — 1,202 session‑generated teacher drawings selected with the official six‑aspect rubric plus the 30 provided public examples. Target format is the harmony final channel only.
Validation (held out, rebuilt byte‑for‑byte by code/build_clean_eval.py): on the 24 held‑out instructions the submitted checkpoint produced a complete, schema‑clean SVG inside the 32,768‑byte budget for 20 (8 simple / 7 medium / 5 complex) against 18 for round 1; every invalid answer is a repetition loop truncated at the generation limit, never a schema violation.
Rejected variant, documented: a fourth round on a heavier complex mixture (416 steps, 1,650 rows) tied on validity (20/24, same 17/19 on items unseen in any round), was behind on the pixel proxy, and was worse on the medium stratum in side‑by‑side inspection, so its weights were not installed — its log, answers and contact sheets are in the evidence.
Serving check (logs/selfcheck_final.json): static review/evidence/weight‑audit gates passed, startup 107 s of 900 s, speed gate 436.7 s of 10,800 s, 16/16 answers parsed, one invalid public answer (a per‑item zero, not a request failure).
Not obtained: the gpt‑6‑sol judge endpoint returned HTTP 409 provider_rejected for the entire final hours (60+ probes, three watchers), so the final checkpoints could not be scored with the rubric itself; the comparison rests on grader‑exact validity checks, pixel distance and side‑by‑side rendering.