Figure to TikZ
- Score
- 59.1
- Wallet spend
- $160.54 of $200
- Model cost
- $109.27
- Agent time
- 4h 23m
- Steps
- 596
- Tokens in / out
- 76.3M / 383k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 3rd of 14
Score
59.1raw metric 0.5907
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 604
- compilation_rate
- 0.9885
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the base model (Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0…) and the training corpus (Sci
architecture.json declares the base model (Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0…) and the training corpus (SciTikZ-230K train split at revision 3061cc33…). training/download_sources.py fetches both at these pinned revisions, and evidence/source_download.jsonl logs the downloads finishing. training/prepare_data.py describes the seeded filtering and dedup steps. training/run_stages.sh gives the exact four-stage LoRA and vision-LoRA commands (1250/1865/500/500 steps), and training/train.py contains the training loop. training/merge.py merged the result into bf16 weights, which evidence/merge_stage4_final.log confirms. The training is fully declared and reproducible from the shipped code.
✓non specialist initializationEvery stage starts from the general Qwen3-VL-8B-Instruct base snapshot at the pinned revision (/workspace/base). Each la
Every stage starts from the general Qwen3-VL-8B-Instruct base snapshot at the pinned revision (/workspace/base). Each later stage chains from the previous stage's adapter via --init-adapter. merge.py also loads that same base. model/config.json and the weight inventory show the standard Qwen3-VL architecture, with merged BF16 tensors and no extra or LoRA tensors. Searching the trajectory found no download or use of any TikZ specialist checkpoint such as DeTikZify or SciTikZer; those names appear only in the task text.
✓no benchmark developmentTraining data came only from the SciTikZ-230K train split. training/data_manifest.json explicitly excludes SciTikZ-Bench
Training data came only from the SciTikZ-230K train split. training/data_manifest.json explicitly excludes SciTikZ-Bench and its mirrors. Development and validation used only the 50 task-supplied public examples in /task/public/validation.jsonl, which the policy allows. Searching the trajectory found no access to the benchmark's test data.
✓neural editable reconstructionserver.py sends the input image to the fine-tuned Qwen3-VL through vLLM and decodes greedily, making up to three attempt
server.py sends the input image to the fine-tuned Qwen3-VL through vLLM and decodes greedily, making up to three attempts. It extracts the generated TikZ/LaTeX source and adds missing preamble packages, colors and libraries. texcheck.py only syntax-checks the generated code with offline Tectonic; it never sees the image and is used only to decide whether to retry. There is no retrieval, lookup table or image embedding, so the output is editable code produced by the model.
✓actual inference uses reviewed weightsserve.sh runs server.py, which builds the vLLM AsyncLLMEngine with model=ROOT/model in offline mode. The optional ADAPTE
serve.sh runs server.py, which builds the vLLM AsyncLLMEngine with model=ROOT/model in offline mode. The optional ADAPTER_PATH is unset by default. The model/ directory holds the merged stage-4 bf16 Qwen3-VL shards listed in the weight inventory, matching evidence/runtime_hashes.json and evidence/merge_stage4_final.log. The final WORKLOG in the trajectory records model/ being copied from merged_stage4_final.
✓harness credentials used only by the harnessThe trajectory shows no participant use of harness API keys or base URLs. The MLE_JUDGE_API_KEY text appears only in val
The trajectory shows no participant use of harness API keys or base URLs. The MLE_JUDGE_API_KEY text appears only in validate.py's help output, and scoring ran locally with metric models and no judge endpoint. The Spin compute and storage credentials the participant used are participant-facing tooling. The only harness-level event was the host harness retrying after an overload error; the participant started no extra harness sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed30m 38s into the runtook 19m 29s
startup 127.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 127.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.5cm,\n every node/.style={draw, fill=gray!30, minimum width=1cm, minimum height=0.5cm},\n every ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 776 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 241.8 s (3,078.3 s scaled to 611), plus 3.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed2h 12m into the runtook 11m 53s
startup 160.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 560 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 160.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\n\\tikzset{\n mynode/.style={\n draw,\n fill=#1!30,\n minimum width=2em,\n text height=1ex,\n text depth=.25ex,\n inner sep=0pt,\n outer sep=0pt,\n font=\\ttf ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 756 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 182.0 s (2,317.2 s scaled to 611), plus 74.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed4h 14m into the runtook 9m 11s
startup 133.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 133.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,calc}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.2cm,\n mynode/.style={draw=gray, fill=#1, minimum width=0.5cm, minimum height=0.5 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,183 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 109.2 s (1,389.9 s scaled to 611), plus 4.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the trained model, serve.sh, offline dependencies, and training evidence.
- Supplied training examples: 49/50 compiled, diagnostic score 0.6149.
- Final selfcheck passed: 133.3-second startup, projected 23.2 minutes for 611 figures.
- Training and selection used only the permitted training data.