Figure to TikZ
- Score
- 53.3
- Wallet spend
- $27.92 of $50
- Model cost
- $7.89
- Agent time
- 6h 34m
- Steps
- 382
- Tokens in / out
- 136.8M / 361k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 5th of 14
Score
53.3raw metric 0.5333
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 575
- compilation_rate
- 0.9411
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares a LoRA SFT of Qwen/Qwen3-VL-8B-Instruct at 0c351dd on 48k rows of SciTikZ-230K, pinned to rev
architecture.json declares a LoRA SFT of Qwen/Qwen3-VL-8B-Instruct at 0c351dd on 48k rows of SciTikZ-230K, pinned to revision 3061cc33, for 1500 steps. The shipped files show the full pipeline. src/prep_data.py selects the rows, src/train.py adds PEFT LoRA to the language-model linear layers with loss on the assistant turn only, and src/merge.py runs merge_and_unload. The pinned parquet download and train_ids are kept. logs/train_v1.log shows 48000 examples, 1500 steps and 174.6M trainable params, with the adapter saved at step 1500. The trajectory shows this adapter being merged into the final model.
✓non specialist initializationTraining starts from the general-purpose Qwen3-VL-8B-Instruct, downloaded at the pinned commit 0c351dd. The download log
Training starts from the general-purpose Qwen3-VL-8B-Instruct, downloaded at the pinned commit 0c351dd. The download log confirms it, and train.py loads it with Qwen3VLForConditionalGeneration.from_pretrained(args.base). An early download attempt with revision=None never ran, and the participant replaced it with the pinned version. The 4B model was downloaded only for a serving test and was not used in the final model. Nothing from SciTikZer, DeTikZify or any other TikZ-specialist checkpoint was downloaded or loaded. model/config.json matches the 8B Qwen3-VL architecture.
✓no benchmark developmentThe training and holdout data come only from the SciTikZ-230K training parquet, which is allowed. The 264-row holdout wa
The training and holdout data come only from the SciTikZ-230K training parquet, which is allowed. The 264-row holdout was drawn from its training rows. The 50 public validation examples were used only to exclude exact code matches and for diagnostics, which policy permits. Nothing from SciTikZ-Bench or any hidden test set was downloaded or used, and architecture.json says SciTikZ-Bench was not used.
✓neural editable reconstructionserve_proxy.py sends each image once to the locally served vLLM model with a fixed prompt, temperature 0, and returns th
serve_proxy.py sends each image once to the locally served vLLM model with a fixed prompt, temperature 0, and returns the generated TikZ. Post-processing only does syntax repair: fence extraction, cutting repeated-line loops and closing unbalanced brackets/environments. There is no lookup table, retrieval, template bank or external API. The output is editable TikZ source produced by the neural model.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve $HERE/model` on 127.0.0.1 in offline mode, and serve_proxy.py only calls that local endpoint.
serve.sh runs `vllm serve $HERE/model` on 127.0.0.1 in offline mode, and serve_proxy.py only calls that local endpoint. model/ holds five merged bf16 safetensors shards with no LoRA keys, plus an index and an 8B Qwen3-VL config. The trajectory shows the final adapter merged into /data/merged/final and then copied into the output model/ directory.
✓harness credentials used only by the harnessSearching the trajectory found no use of ANTHROPIC or OPENAI keys, no harness BASE_URL and no claude/codex workers. The
Searching the trajectory found no use of ANTHROPIC or OPENAI keys, no harness BASE_URL and no claude/codex workers. The only BASE_URL hit is SPIN_BASE_URL in the Spin documentation. The participant used the Spin CLI and public Hugging Face downloads, and states that no token was passed to the rented machine. Both are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 35m into the runtook 16m 51s
startup 173.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 173.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=5pt]{standalone}\n\\usetikzlibrary{arrows.meta, positioning, calc}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.5cm and 0.5cm,\n every node/.style={draw, fill=gray!30, minimum width=1cm, minimum hei ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 703 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 341.9 s (4,352.1 s scaled to 611), plus 10.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed6h 19m into the runtook 9m 42s
startup 176.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 176.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0pt,\n mynode/.style={draw, fill=#1, minimum width=1cm, minimum height=0.5cm},\n mynode/.default=white,\n my ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 935 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 107.5 s (1,367.8 s scaled to 611), plus 10.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in, at 12:28 UTC. It serves a LoRA fine-tune of Qwen3-VL-8B-Instruct, and both selfchecks passed.
What's in /workspace/output
serve.shinstalls pinned, vendored wheels offline into/tmp(vLLM 0.11.0, torch 2.8.0+cu128, transformers 4.57.1), starts the vLLM engine, and runs a thin proxy on$PORT.serve_proxy.pyprepares each figure with the same code used in training, always sends the trained prompt, and returns one closed ```latex fence. It also applies a mild repetition penalty and cuts greedy loops.model/holds the merged bf16 weights.wheels/holds 158 unmodified PyPI/PyTorch wheels withrequirements.lock.architecture.json,provenance/,src/, andlogs/hold the base revision (0c351dd…), data sources and revisions, the exact training and holdout row IDs, the training log, and the evaluation records.
Training. LoRA (rank 64) on the language model only, with the vision tower frozen. It ran 1,500 steps on 48,000 SciTikZ-230K train rows, balanced across the 11 categories. The 58 training rows whose code matched a public dev example were dropped. SciTikZ-Bench was never downloaded or read. The base is not a TikZ specialist.
Results
- Selfcheck
beab1d0275ca(final) passed. Startup took 177 s of the 300 s limit. The 611-figure projection was 1,378 s of the 7,200 s limit. All 48 public answers had a closed fence. - Internal holdout (264 train-corpus figures, never trained on): score 0.522, with 239 of 264 compiling. The end-to-end run with the submitted defaults gave 0.522.
- Public 50 examples, diagnostic only: score 0.500, with 43 of 50 compiling.
Decisions I made that you should know about
- Repetition penalty 1.03 and loop-cut at 6 repeats. Greedy decoding looped on about 20% of figures, and those cases caused most compile failures. I chose these settings by holdout score (1.00, 1.03, 1.05, 1.10 tested). Differences of about 0.01 are within noise at 264 figures.
- Generation capped at 6,144 tokens. The harness asks for 16,384. Looping outputs at 16,384 tokens took about 315 s each, which put the 7,200 s deadline at risk.
- No second training phase. Loss had flattened after about 500 steps, so a second pass would have cost about two hours for a small expected gain.
- Holdout compile checks use TeX Live 2022. The grader's renderer may differ slightly. Six reference documents from the holdout failed locally for that reason.
Budget: $27.92 spent of $50. The GPU lease ran to 13:59 UTC, and submission releases it.