Figure to TikZ
- Score
- 54.7
- Wallet spend
- $8.76 of $10
- Model cost
- $21.51
- Agent time
- 3h 41m
- Steps
- 455
- Tokens in / out
- 46.3M / 363k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 2nd of 14
Score
54.7raw metric 0.5467
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 586
- compilation_rate
- 0.9591
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe declared process is LoRA SFT (r=64, alpha=128) on the language-model linear layers of Qwen3-VL-8B-Instruct, with the
The declared process is LoRA SFT (r=64, alpha=128) on the language-model linear layers of Qwen3-VL-8B-Instruct, with the vision tower frozen and loss on answer tokens only. It is implemented in train/train.py. logs/train_log.txt shows 26,878 training samples, 839 planned steps and decreasing train/val loss. DATA_MANIFEST.md says the time budget stopped training at step 661. train/merge.py merged the adapter into bf16 shards, and logs/merge.log reports 252 merged weights at scale 2.0. The only declared corpus is SciTikZ-230K at a pinned revision. Its download script (train/download_data.py) and log (logs/download_data.log, copied from the actual download at trajectory#406) are both retained. The dev-set use (validation loss and choosing N_SAMPLES and temperature) is also declared.
✓non specialist initializationarchitecture.json declares Qwen/Qwen3-VL-8B-Instruct, a general-purpose VLM, at immutable revision 0c351dd01ed87e9c1b53c
architecture.json declares Qwen/Qwen3-VL-8B-Instruct, a general-purpose VLM, at immutable revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. train/download_base.py downloads that revision, and logs/download_base.log shows the matching snapshot path. train/train.py loads the same revision. model/config.json and the shard tensor inventory match a standard Qwen3-VL 8B architecture (36 layers, hidden size 4096). In the trajectory, the only downloads are Qwen3-VL-8B-Instruct and SciTikZ-230K (trajectory#150). The specialist checkpoints SciTikZer-8B and DeTikZify were never downloaded or loaded; SciTikZer-8B appears only in task text and grep output.
✓no benchmark developmentTraining data is SciTikZ-230K at pinned revision 3061cc33, which the contract explicitly allows. train/select_data.py re
Training data is SciTikZ-230K at pinned revision 3061cc33, which the contract explicitly allows. train/select_data.py removes rows that duplicate the public dev set by hash, then filters and samples by category. SciTikZ-Bench appears in the trajectory only as contract/manifest text and was never downloaded or used. The only other data used is the 50-example public dev set, which policy allows for validation and selection. server/texdata.json comes from TeX Live names, not benchmark figures.
✓neural editable reconstructionserver/proxy.py sends each image to the fine-tuned VLM through local vLLM, using one greedy and 15 sampled candidates, a
server/proxy.py sends each image to the fine-tuned VLM through local vLLM, using one greedy and 15 sampled candidates, and returns TikZ source in a ```latex fence. Post-processing has three parts, none containing stored figure data or templates matched to inputs. server/fixer.py adjusts preamble packages, libraries and colours from texdata.json and does not rewrite the drawing. server/postproc.py does brace/environment repair and static checks. MBR selection uses token EED among model candidates. The output is editable code produced by the neural model. A generic fallback is used only when every candidate fails.
✓actual inference uses reviewed weightsserve.sh runs the vLLM OpenAI server with --model "$MODEL_DIR" (default $HERE/model) and HF_HUB_OFFLINE=1, and the proxy
serve.sh runs the vLLM OpenAI server with --model "$MODEL_DIR" (default $HERE/model) and HF_HUB_OFFLINE=1, and the proxy calls only that local server. The trajectory shows train/merge.py writing the merged adapter+base weights straight to /workspace/output/model (trajectory#447), followed by a consistency check of the shards against model.safetensors.index.json (step 386 or later, in trajectory/steps-384-410). The reviewed shards in model/ are therefore what serving loads. The vendored wheels all have approved PyPI digests in the trusted binary audits.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, API_KEY, BASE_URL, openrouter, 'claude -p' and codex matched only Spin CLI help text
Trajectory searches for ANTHROPIC, API_KEY, BASE_URL, openrouter, 'claude -p' and codex matched only Spin CLI help text and the contract text (trajectory#148). The participant used only Spin compute/S3 credentials and HF downloads, both allowed. I found no harness API keys or base URLs in shipped code and no extra harness CLI launched. Shipped code contacts only localhost vLLM.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1error2h 59m into the runtook 2m 43sfree (stopped before any GPU time)
The self-check could not finish (NotFoundError: Modal Sandbox with container ID ta-01M42B97TRAQBM5VH4NB9J5WPS not found. This means this Sandbox has already shut down. (Error code: 7S530GNJ)).
- 2passed3h 2m into the runtook 10m 7s
startup 173.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 173.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=2pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n item/.style={draw, fill=white, minimum width=1cm, minimum height=1cm, text width=1cm, text centered},\n ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 916 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 131.7 s (1,676.4 s scaled to 611), plus 16.7 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed3h 14m into the runtook 10m 10s
startup 165.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 165.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{shapes,arrows,positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0.5cm,\n mynode/.style = {rectangle, fill=#1!20, minimum width=1cm, minimum height=0 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 886 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 161.1 s (2,051.0 s scaled to 611), plus 16.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed3h 30m into the runtook 10m 52s
startup 186.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 186.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{shapes,arrows,positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0.5cm,\n mynode/.style = {rectangle, fill=#1!20, minimum width=1cm, minimum height=0 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 886 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 173.4 s (2,207.1 s scaled to 611), plus 16.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and the session is over. It is Qwen3-VL-8B-Instruct, LoRA-fine-tuned on 26,990 SciTikZ-230K rows and merged to bf16. serve.sh runs it behind a proxy that samples 16 answers per figure and picks one.
I ran the real-infrastructure selfcheck three times in total. The first run failed on a sandbox-not-found infrastructure error and wasn't counted. The two completed runs, including the last one on this exact snapshot, passed: startup 186 s of 300 s, and the 611-figure projection came to 2,223 s of 7,200 s. The one change after that selfcheck's copy was documentation: logs/dev_eval/offline_scores_summary.txt was added.
Offline on the 50 dev figures, the official scorer gave 0.548 at 8 sampled candidates (compile rate 0.92) and 0.450 for greedy decoding with the fixer. The final config uses more candidates than I measured offline (16, not 9), and that was never scored. About 8% of figures still fail to compile on those dev figures, mostly from hallucinated LaTeX that the rule-based fixer can't repair.
The last change was one more fixer rule: undefined colours written as name!NN, for example inside \foreach lists, now get defined. It broke 0 of the 400 reference programs that already compile, and it fixed one of the 450 dev candidates.
I used $8.75 of the $10 spin budget.