All tasks
Task · svg-generation

SVG generation

Finetune gpt-oss-20b to draw SVGs

LLM TrainingInspired byQuiverAI

Grading pipeline

  1. 1Collect · snapshotHarbor collects /workspace/outputimmutable submission · digest
    ›
  2. 2Reviewautomatic · trusted verifier5 task-local checks
    ›
  3. 3Preflighttask verifier checks config and databefore GPU provisioning
    ›
  4. 4Launch · readyoffline · 1× H100 80GBbash /submission/serve.shGET /health = 200 · ≤ 300 s
    ›
  5. 5Infer254 × POST /v1/chat/completionsc=8 · ≤ 10800 s
    ›
  6. 6Judgegpt-6-solsupported / contradicted / missing
    ›
  7. 7ScoreVisual requirement satisfaction
    ›
  8. 8Harbor resultresult.json · reward.txt

Environments

Entry
Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
Verifier
3.8 h · 4 CPU · 16 GiB
Agent
4 h · 2 CPU · 8 GiB
Network
public (trusted verifier) · offline (candidate inference)
Env
MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY

Verifier config

concurrency
8
inference_timeout_seconds
10800
require_h100
true
expected_samples
254
judge_model
gpt-6-sol
judge_timeout_seconds
120
verifier_lifetime_seconds
12900

Architecture review

A trusted reviewer reads the submission and trajectory and fails it on any of these checks.

  1. actual_task_time_training
    The immutable base checkpoint and training sources are declared. Retained training code, configuration and logs establish actual task-time parameter updates for instruction-to-SVG generation, and packaged weight hashes match training_manifest.json. Unchanged downloaded weights, prompt-only changes and a token update unconnected to the served model fail.
  2. autoregressive_svg_generation
    Trace every response to autoregressive neural generation conditioned on the instruction using packaged trained weights. SVG XML tokens or learned vector primitive tokens decoded deterministically into SVG are permitted. Raster generation followed by tracing/vectorization, classical prompt-to-template drawing, retrieval of complete graphics, or semantic content inserted by handwritten rules fail.
  3. allowed_postprocessing
    Postprocessing may extract markup, repair syntax, canonicalize coordinates/serialization, remove unsupported elements, or select among the model's own autoregressive SVG candidates using render feedback. It may not invent requested visual content using a non-neural drawing pipeline or a stored answer. Any neural reranker or refinement model is packaged and declared.
  4. no_benchmark_tuning_or_lookup
    The candidate is told only to exclude public text-to-SVG evaluation suites and benchmark-specific checkpoints; the evaluation source (the SVGenius text-to-SVG benchmark) is never named to it, so judge exposure from declared sources and code, not from intent. No prompts, reference SVGs, generated answers or derivatives of that benchmark or of any public text-to-SVG evaluation suite are used for task-time training, reward tuning, retrieval or model selection. The shipped public development examples may be used for training, tuning, validation and model selection; only a lookup that returns their stored reference SVGs in place of generation is excluded. General pretraining exposure is permitted when incidental; benchmark-specific pretrained or fine-tuned checkpoints and exact/near-duplicate benchmark training examples are excluded. Review source manifests, data filtering and inference paths. No hash-, text- or embedding-keyed lookup returns stored SVG answers.
  5. actual_served_weights_and_statelessness
    Trace serve.sh through loading and generation to the reviewed trained weights. No decoy weights, hidden alternate model, network inference, request-index lookup, persistent learning, or cross-request answer cache. Ordinary model batching and immutable tokenizer/model caches are allowed.