All tasks
Task · svg-generation
SVG generation
Finetune gpt-oss-20b to draw SVGs
LLM TrainingInspired by
QuiverAI
Grading pipeline
Environments
- Entry
- Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
- Verifier
- 3.8 h · 4 CPU · 16 GiB
- Agent
- 4 h · 2 CPU · 8 GiB
- Network
- public (trusted verifier) · offline (candidate inference)
- Env
- MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY
Verifier config
- concurrency
- 8
- inference_timeout_seconds
- 10800
- require_h100
- true
- expected_samples
- 254
- judge_model
- gpt-6-sol
- judge_timeout_seconds
- 120
- verifier_lifetime_seconds
- 12900
Architecture review
A trusted reviewer reads the submission and trajectory and fails it on any of these checks.
- actual_task_time_trainingThe immutable base checkpoint and training sources are declared. Retained training code, configuration and logs establish actual task-time parameter updates for instruction-to-SVG generation, and packaged weight hashes match training_manifest.json. Unchanged downloaded weights, prompt-only changes and a token update unconnected to the served model fail.
- autoregressive_svg_generationTrace every response to autoregressive neural generation conditioned on the instruction using packaged trained weights. SVG XML tokens or learned vector primitive tokens decoded deterministically into SVG are permitted. Raster generation followed by tracing/vectorization, classical prompt-to-template drawing, retrieval of complete graphics, or semantic content inserted by handwritten rules fail.
- allowed_postprocessingPostprocessing may extract markup, repair syntax, canonicalize coordinates/serialization, remove unsupported elements, or select among the model's own autoregressive SVG candidates using render feedback. It may not invent requested visual content using a non-neural drawing pipeline or a stored answer. Any neural reranker or refinement model is packaged and declared.
- no_benchmark_tuning_or_lookupThe candidate is told only to exclude public text-to-SVG evaluation suites and benchmark-specific checkpoints; the evaluation source (the SVGenius text-to-SVG benchmark) is never named to it, so judge exposure from declared sources and code, not from intent. No prompts, reference SVGs, generated answers or derivatives of that benchmark or of any public text-to-SVG evaluation suite are used for task-time training, reward tuning, retrieval or model selection. The shipped public development examples may be used for training, tuning, validation and model selection; only a lookup that returns their stored reference SVGs in place of generation is excluded. General pretraining exposure is permitted when incidental; benchmark-specific pretrained or fine-tuned checkpoints and exact/near-duplicate benchmark training examples are excluded. Review source manifests, data filtering and inference paths. No hash-, text- or embedding-keyed lookup returns stored SVG answers.
- actual_served_weights_and_statelessnessTrace serve.sh through loading and generation to the reviewed trained weights. No decoy weights, hidden alternate model, network inference, request-index lookup, persistent learning, or cross-request answer cache. Ordinary model batching and immutable tokenizer/model caches are allowed.