Figure to TikZ
- Score
- 32.9
- Wallet spend
- $34.52 of $200
- Model cost
- $1.34
- Agent time
- 6h 34m
- Steps
- 1,018
- Tokens in / out
- 119.1M / 199k
- Self-checks
- 1 passed of 2 used
- Rank at $200 wallet
- 13th of 14
Score
32.9raw metric 0.3292
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 421
- compilation_rate
- 0.6890
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json names Qwen/Qwen2.5-VL-3B-Instruct as the base model, pinned to revision 66285546d2b821cf421d4f5eb25763
architecture.json names Qwen/Qwen2.5-VL-3B-Instruct as the base model, pinned to revision 66285546d2b821cf421d4f5eb2576359d3770cd3. train.py and train.sh run a BF16 LoRA SFT (rank 16, alpha 32, all-linear) for image-to-TikZ generation, then merge the adapter into model/. training_manifest.json records the data and method. logs/training.log shows 20,000 SciTikZ-230K rows plus 48 supplied examples, the base model loaded at the pinned revision, losses over 2506 steps, the LoRA merge and the final save. The trajectory shows train.sh/train.py running on a Spin machine (trajectory#128) and the resulting model and adapter being uploaded and then copied into the output folder (trajectory#967, trajectory#968, trajectory#970). These are trained weights, not unmodified ones.
✓non specialist initializationtrain.py loads Qwen2_5_VLForConditionalGeneration from Qwen/Qwen2.5-VL-3B-Instruct at the pinned revision, a general mul
train.py loads Qwen2_5_VLForConditionalGeneration from Qwen/Qwen2.5-VL-3B-Instruct at the pinned revision, a general multimodal model. adapter_config.json gives the same base_model_name_or_path. model/config.json and the safetensors inventory match the Qwen2.5-VL-3B layout. The trajectory never mentions SciTikZer, DeTikZify or TikZero as a download or starting point; SciTikZer appears only in the task prompt (trajectory#1).
✓no benchmark developmentTraining data is JackieLin0123/SciTikZ-230K, train split only, at revision 3061cc331aa6f71cc7430e0480929d0ccddade67, plu
Training data is JackieLin0123/SciTikZ-230K, train split only, at revision 3061cc331aa6f71cc7430e0480929d0ccddade67, plus the 50 organizer-provided public examples. provided50.tar.gz was packed directly from /task/public validation.jsonl and its images (trajectory#24-54 region), and the organizer says those examples are permitted SciTikZ-230K samples. The download command is kept in train.py and training_manifest.json. The trajectory shows no download or use of SciTikZ-Bench; the name appears only in task instructions.
✓neural editable reconstructionserver.py forwards the chat request, which carries the image, to a local vLLM server running the merged Qwen2.5-VL model
server.py forwards the chat request, which carries the image, to a local vLLM server running the merged Qwen2.5-VL model. It only post-processes the generated text to pull out or wrap the ```latex fence, falling back to an empty tikzpicture when output is empty. There is no raster embedding, tracing, OCR or answer lookup.
✓actual inference uses reviewed weightsserve.sh installs local wheels offline with HF_HUB_OFFLINE set, then runs server.py. server.py starts vllm.entrypoints.o
serve.sh installs local wheels offline with HF_HUB_OFFLINE set, then runs server.py. server.py starts vllm.entrypoints.openai.api_server with --model ROOT/model on 127.0.0.1 and proxies requests to it. model/ holds the merged Qwen2.5-VL-3B safetensors shards, which the trajectory shows were produced by the training run and copied from the run's storage (trajectory#967, trajectory#968, trajectory#969). No external model or service is called.
✓harness credentials used only by the harnessSearching the trajectory found no OPENAI, ANTHROPIC, API_KEY or BASE_URL references, no curl calls, no extra codex/claud
Searching the trajectory found no OPENAI, ANTHROPIC, API_KEY or BASE_URL references, no curl calls, no extra codex/claude CLI sessions and no printenv. External access went through Spin compute and Spin S3 transfer sessions (trajectory#129, trajectory#968), which are the participant's own allowance.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed6h 19m into the runtook 7m 49s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 0 mdurl-0.1.2 mistral_common-1.12.0 mpmath-1.3.0 msgpack-1.2.3 msgspec-0.22.0 multidict-6.9.1 nest
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 0 mdurl-0.1.2 mistral_common-1.12.0 mpmath-1.3.0 msgpack-1.2.3 msgspec-0.22.0 multidict-6.9.1 nest_asyncio-1.6.0 networkx-3.7 ninja-1.13.2 numba-0.61.2 numpy-2.2.6 nvidia-cublas-cu12-12.8.3.14 nvidia-cuda-cupti-cu12-12.8.57 nvidia-cuda-nvrtc-cu12-12.8.61 nvidia-cuda-runtime-cu12-12.8.57 nvidia-cudnn-cu12-9.7.1.26 nvidia-cufft-cu12-11.3.3.41 nvidia-cufile-cu12-1.13.0.11 nvidia-curand-cu12-10.3.9.55 nvidia-cusolver-cu12-11.7.2.55 nvidia-cusparse-cu12-12.5.7.53 nvidia-cusparselt-cu12-0.6.3 nvidia-nccl-cu12-2.26.2 nvidia-nvjitlink-cu12-12.8.61 nvidia-nvtx-cu12-12.8.55 openai-1.90.0 opencv-python-headless-5.0.0.93 opentelemetry-api-1.45.1 opentelemetry-exporter-http-transport-0.66b1 opentelemetry-exporter-otlp-common-0.66b1 opentelemetry-exporter-otlp-proto-common-1.45.1 opentelemetry-exporter-otlp-proto-http-1.45.1 opentelemetry-proto-1.45.1 opentelemetry-sdk-1.45.1 opentelemetry-semantic-conventions-0.66b1 outlines-0.1.11 outlines_core-0.1.26 packaging-26.3 partial-json-parser-0.2.1.1.post7 pillow-12.3.0 prometheus-fastapi-instrumentator-8.1.0 prometheus_client-0.26.0 propcache-0.5.4 protobuf-7.36.2 psutil-7.2.2 py-cpuinfo-9.0.0 pybase64-1.5.1 pycountry-26.2.16 pydantic-2.14.0 pydantic-core-2.50.0 pydantic-extra-types-2.11.1 pydantic-settings-2.15.0 pygments-2.21.0 python-dotenv-1.2.4 python-json-logger-4.2.0 python-multipart-0.0.32 pyyaml-6.0.3 pyzmq-27.2.0 ray-2.59.0 referencing-0.37.0 regex-2026.9.29 requests-2.34.2 rich-15.0.0 rich-toolkit-0.20.6 rignore-0.8.1 rpds-py-2026.9.1 safetensors-0.8.0 scipy-1.18.1 sentencepiece-0.2.2 sentry-sdk-2.71.0 setuptools-79.0.1 shellingham-1.5.4 six-1.17.0 sniffio-1.3.1 starlette-1.7.0 sympy-1.14.0 tiktoken-0.14.0 tokenizers-0.21.1 torch-2.7.0+cu128 torchaudio-2.7.0+cu128 torchvision-0.22.0+cu128 tqdm-4.70.1 transformers-4.52.4 triton-3.3.0 ...[clipped]
- 2passed6h 28m into the runtook 6m 24s
startup 185.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 185.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0pt,\n every node/.style = {draw, fill=#1, minimum width=2em},\n every label/.style = {font=\\small, inner se ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 765 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 43.2 s (549.4 s scaled to 611), plus 5.1 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The optimized selfcheck passed on an H100: readiness in 185.4 seconds, all 48 requests returned closed LaTeX fences, and the projected 611-figure generation time was 554.5 seconds at concurrency 16. I’m doing the final submission review now.