Figure to TikZ
- Score
- 35.5
- Wallet spend
- $11.88 of $50
- Model cost
- $24.44
- Agent time
- 7h 5m
- Steps
- 225
- Tokens in / out
- 30.0M / 61k
- Self-checks
- 2 passed of 4 used
- Rank at $50 wallet
- 13th of 14
Score
35.5raw metric 0.355
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 413
- compilation_rate
- 0.6759
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json and DATA_MANIFEST.json name the base model as Qwen/Qwen3-VL-8B-Instruct at rev 0c351dd0… and the datas
architecture.json and DATA_MANIFEST.json name the base model as Qwen/Qwen3-VL-8B-Instruct at rev 0c351dd0… and the dataset as SciTikZ-230K at rev 3061cc33…. download_data.sh pins both revisions. train.py runs LoRA SFT on q/k/v/o/gate/up/down with loss on assistant tokens only. logs/train.log shows pass 1 on 40,000 examples. logs/train2.log shows pass 2, which resumed that adapter and ran on 12,000 longer examples. The trajectory records both passes. One minor gap: the shipped train.py does not have the resume-adapter lines used for pass 2. The trajectory shows those lines were added to a /tmp copy. The training is still declared and evidenced.
✓non specialist initializationThe base is the general-purpose Qwen3-VL-8B-Instruct, downloaded from HF at the pinned revision (download_data.sh; traje
The base is the general-purpose Qwen3-VL-8B-Instruct, downloaded from HF at the pinned revision (download_data.sh; trajectory#161). The adapter inventory contains only LoRA A/B tensors with rank-64 shapes on Qwen3-VL language-model layers, which fits training from that base. There is no evidence of initialization from SciTikZer, DeTikZify, TikZero or any other TikZ specialist.
✓no benchmark developmentprepare_subset.py and prepare_longer.py draw only from the SciTikZ-230K train parquet. DATA_MANIFEST.json and logs/subse
prepare_subset.py and prepare_longer.py draw only from the SciTikZ-230K train parquet. DATA_MANIFEST.json and logs/subset_stats.json mark SciTikZ-Bench as not used, and trajectory searches found no Bench download. The only development evaluation used the 50 public SciTikZ-230K examples and self-checks on public inputs, which the organizers allow.
✓neural editable reconstructionserver.py gives the image to Qwen3-VL as a vision prompt and greedily generates TikZ. repair.py then does only syntax fi
server.py gives the image to Qwen3-VL as a vision prompt and greedily generates TikZ. repair.py then does only syntax fixes: it adds a preamble, closes environments and braces, and drops lines that fail pdflatex. Nothing traces pixels, embeds the raster, runs OCR or looks up stored answers.
✓actual inference uses reviewed weightsserve.sh runs server.py offline. server.py loads the shipped base/ with local_files_only, applies the shipped adapter/ t
serve.sh runs server.py offline. server.py loads the shipped base/ with local_files_only, applies the shipped adapter/ through PeftModel, merges it and generates with the result. The trajectory shows adapter/adapter_model.safetensors was copied from the trained lora2/final output (trajectory#155, trajectory#160) and base/ was downloaded at the pinned revision (trajectory#161).
✓harness credentials used only by the harnessTrajectory searches for OPENAI, ANTHROPIC and API_KEY found nothing, so harness credentials were not used. The participa
Trajectory searches for OPENAI, ANTHROPIC and API_KEY found nothing, so harness credentials were not used. The participant used only allowed credentials: HF_TOKEN for downloads, plus Spin CLI and S3 for compute and file transfer. The shipped serve.sh and server.py make no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed6h 2m into the runtook 10m 51s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 1.6.0 huggingface-hub-0.35.3 jinja2-3.1.6 markupsafe-3.0.3 mpmath-1.3.0 networkx-3.7 nvidia-cublas
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 1.6.0 huggingface-hub-0.35.3 jinja2-3.1.6 markupsafe-3.0.3 mpmath-1.3.0 networkx-3.7 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.3 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvtx-cu12-12.8.90 packaging-26.3 peft-0.17.1 protobuf-7.36.2 psutil-7.2.2 pydantic-2.13.5 pydantic-core-2.46.5 pyyaml-6.0.3 regex-2026.9.29 sentencepiece-0.2.2 setuptools-79.0.1 sniffio-1.3.1 starlette-1.7.0 sympy-1.14.0 tokenizers-0.22.1 torch-2.8.0 torchvision-0.23.0 tqdm-4.70.1 transformers-4.57.1 triton-3.4.0 typing-inspection-0.4.4 uvicorn-0.54.0 Traceback (most recent call last): File "/submission/server.py", line 16, in <module> from fastapi import Body, FastAPI File "/opt/venv/lib/python3.12/site-packages/fastapi/__init__.py", line 7, in <module> from .applications import FastAPI as FastAPI File "/opt/venv/lib/python3.12/site-packages/fastapi/applications.py", line 7, in <module> from fastapi import routing File "/opt/venv/lib/python3.12/site-packages/fastapi/routing.py", line 58, in <module> from fastapi.dependencies.utils import ( File "/opt/venv/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 52, in <module> from fastapi.background import BackgroundTasks File "/opt/venv/lib/python3.12/site-packages/fastapi/background.py", line 5, in <module> from fastapi.telemetry._api import _operation File "/opt/venv/lib/python3.12/site-packages/fastapi/telemetry/__init__.py", line 1, in <module> from ._api import TelemetryConfig as TelemetryConfig File "/ ...[clipped]
- 2failed6h 17m into the runtook 9m 8s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidi
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.3 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvtx-cu12-12.8.90 packaging-26.3 peft-0.17.1 protobuf-7.36.2 psutil-7.2.2 pyyaml-6.0.3 regex-2026.9.29 sentencepiece-0.2.2 setuptools-79.0.1 sympy-1.14.0 tokenizers-0.22.1 torch-2.8.0 torchvision-0.23.0 tqdm-4.70.1 transformers-4.57.1 triton-3.4.0 Error importing huggingface_hub.hf_api: No module named 'requests' Traceback (most recent call last): File "/submission/server.py", line 23, in <module> from peft import PeftModel File "/opt/venv/lib/python3.12/site-packages/peft/__init__.py", line 17, in <module> from .auto import ( File "/opt/venv/lib/python3.12/site-packages/peft/auto.py", line 21, in <module> from transformers import ( File "/opt/venv/lib/python3.12/site-packages/transformers/__init__.py", line 27, in <module> from . import dependency_versions_check File "/opt/venv/lib/python3.12/site-packages/transformers/dependency_versions_check.py", line 16, in <module> from .utils.versions import require_version, require_version_core File "/opt/venv/lib/python3.12/site-packages/transformers/utils/__init__.py", line 19, in <module> from huggingface_hub import get_full_repo_name # for backward compatibility ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/huggingface_hub/__init__.py", line 1040, in __getattr__ submod = importlib.import_module(submod_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^ ...[clipped]
- 3passed6h 30m into the runtook 12m 25s
startup 98.5 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 559 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 98.5 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=2pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,fit,calc}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0pt,\n box/.style = {draw, minimum width=1.5em, minimum height=1.5em, outer s ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 4,761 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 288.2 s (3,668.2 s scaled to 611), plus 79.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed6h 48m into the runtook 13m 6s
startup 115.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 558 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 115.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=2pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,fit,calc}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0pt,\n box/.style = {draw, minimum width=1.5em, minimum height=1.5em, outer s ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 4,761 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 337.5 s (4,296.4 s scaled to 611), plus 101.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. It serves a fine-tuned Qwen3-VL-8B that writes TikZ from a figure image.
The base model is Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. Training was LoRA rank 64 on the train split of SciTikZ-230K only: 40,000 shorter figures, then 12,000 longer ones. SciTikZ-Bench was not used.
serve.sh installs the vendored wheels and exposes POST /v1/chat/completions. The model writes the LaTeX. The image is only a vision prompt. A truncated generation is closed so pdfLaTeX has a complete document.
The last self-check passed: startup in 115 seconds of the 300-second limit, and the 611-figure run projects to about 73 minutes of the 2-hour limit. On the 50 public development figures, 28 compiled before that closing repair.