Figure to TikZ
- Score
- 43.0
- Wallet spend
- $31.65 of $200
- Model cost
- $15.27
- Agent time
- 8h 14m
- Steps
- 235
- Tokens in / out
- 44.1M / 142k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 10th of 14
Score
43.0raw metric 0.4296
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 471
- compilation_rate
- 0.7709
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json names the base model as Qwen/Qwen3-VL-4B-Instruct at immutable revision ebb281ec70b05090aa6165b016eac8
architecture.json names the base model as Qwen/Qwen3-VL-4B-Instruct at immutable revision ebb281ec70b05090aa6165b016eac8ec08e71b17. train.py runs a full supervised fine-tune with the vision blocks frozen, training the model to map an image to fenced TikZ/LaTeX. prep_data.py, data_manifest.json and prep_stats.json document the SciTikZ-230K subset. train_main.log (42096 rows, 2631 steps) and train_phase2.log (15962 rows, 998 steps) record both training runs. The trajectory shows the commands: phase 1 started from models/Qwen3-VL-4B-Instruct (trajectory#58), and phase 2 continued from runs/main/final (trajectory#87 built the phase-2 file; the launch is in the steps 171-203 chunk). The phase-2 final checkpoint was uploaded and then copied into output/model (trajectory#220, trajectory#222). Only generation_config.json was patched. The shard sizes match the shipped inventory.
✓non specialist initializationThe starting weights are the general multimodal model Qwen3-VL-4B-Instruct. download_model.py, download_models.log and t
The starting weights are the general multimodal model Qwen3-VL-4B-Instruct. download_model.py, download_models.log and the trajectory (download_model.py was written in the steps 14-39 chunk; phase 1 launched from models/Qwen3-VL-4B-Instruct in trajectory#58) show this was the only starting checkpoint. model/config.json and the safetensors inventory match the Qwen3-VL 4B architecture: hidden size 2560, 36 layers and a 24-layer vision tower. SciTikZer and DeTikZify only appear in the task prompt, paper reading and web search results. Neither was ever downloaded or loaded.
✓no benchmark developmentdata_manifest.json and download_data.py declare SciTikZ-230K, revision 3061cc331aa6f71cc7430e0480929d0ccddade67, as the
data_manifest.json and download_data.py declare SciTikZ-230K, revision 3061cc331aa6f71cc7430e0480929d0ccddade67, as the only corpus. prep_data.py samples only from that parquet file. The trajectory shows the download step and builds the training files only from data_train/train.jsonl (trajectory#56, trajectory#87). The 50 public dev examples were used only as an HTTP diagnostic, which the policy allows (trajectory#144 onwards). No step downloads or uses SciTikZ-Bench.
✓neural editable reconstructionserve.sh starts vLLM serving the fine-tuned Qwen3-VL model, which writes TikZ conditioned on the input image. serve_prox
serve.sh starts vLLM serving the fine-tuned Qwen3-VL model, which writes TikZ conditioned on the input image. serve_proxy.py only caps max_tokens, disables streaming and closes or wraps the latex fence. There is no raster embedding, tracing, OCR, retrieval or lookup.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server with --model $DIR/model, in offline mode (HF_HUB_OFFLINE/TRANSFORMERS_O
serve.sh runs vllm.entrypoints.openai.api_server with --model $DIR/model, in offline mode (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE). serve_proxy.py forwards requests only to the local 127.0.0.1:18001 vLLM server. model/ holds the Qwen3-VL safetensors listed in the inventory, and the trajectory traces those exact files to the phase-2 training output (trajectory#221, trajectory#222). No external services, decoy checkpoints or other backbones were found.
✓harness credentials used only by the harnessNo trajectory step reads, echoes or uses ANTHROPIC/OPENAI/OPENROUTER keys or harness base URLs, and none launches anothe
No trajectory step reads, echoes or uses ANTHROPIC/OPENAI/OPENROUTER keys or harness base URLs, and none launches another agent CLI. The only token passed to the Spin VM was HF_TOKEN (the steps 14-39 chunk), which was used for Hugging Face downloads; that is permitted. Spin storage transfers are the participant's own allowance.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed5h 51m into the runtook 9m 43s
startup 188.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 188.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=3pt]{standalone}\n\\usepackage{tikz}\n\\usepackage{xstring,calc}\n\\usetikzlibrary{calc,positioning}\n\n\\newlength{\\NodeSize}\n\\tikzset{DNA Style/.style={minimum size=0.5cm, draw=gray, line width=1pt, inner sep=2pt}}\n\ ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,398 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 154.6 s (1,968.1 s scaled to 611), plus 18.7 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed7h 54m into the runtook 7m 33s
startup 130.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 130.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[crop, tikz]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\definecolor{mygreen}{HTML}{00FF00}\n\\definecolor{mymauve}{rgb}{0.58,0,0.82}\n\\definecolor{mygold}{HTML}{B8860B}\n\\definecolor{mynavy}{HTML}{000080 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,308 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 139.9 s (1,780.3 s scaled to 611), plus 17.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed8h 3m into the runtook 10m 5s
startup 171.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 171.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[crop, tikz]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\definecolor{mygreen}{HTML}{00FF00}\n\\definecolor{mymauve}{rgb}{0.58,0,0.82}\n\\definecolor{mygold}{HTML}{B8860B}\n\\definecolor{mynavy}{HTML}{000080 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,308 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 111.4 s (1,418.5 s scaled to 611), plus 15.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-13 02:41 UTC.
Model: Qwen/Qwen3-VL-4B-Instruct (ebb281ec…) fine-tuned on SciTikZ-230K train split only (49,562 prepared rows, category-balanced; phase 1: 42,096 rows, lr 3e-5, 2,631 steps; phase 2: remaining 15,962 rows, lr 1e-5, 998 steps; vision blocks frozen, loss on the fenced LaTeX answer). SciTikZ-Bench was never used; the 50 public dev figures were only a post-training HTTP diagnostic.
Verified results:
- Dev diagnostic (VM, pdflatex): 42/50 compiled (84%), vs 41/50 for the phase-1 checkpoint
frontier selfcheckon the exact final folder: PASSED — startup 171.8s/300s, projected 611 figures in 1,434s/7,200s, all 48 batch answers with a closed ```latex fence
Submission (/workspace/output, 185 files, 13.86 GB): fine-tuned weights in model, serve.sh (installs vendored wheels to /tmp, runs vLLM 0.11.0 behind serve_proxy.py, which caps generation at 4,096 tokens to prevent runaway loops), plus provenance in architecture.json, data_manifest.json, TRAINING.md and training logs.
Compute spend: $31.64 of the $200 budget; the H100 lease was stopped before submitting.