Figure to TikZ
- Score
- 39.6
- Wallet spend
- $25.89 of $50
- Model cost
- $1.64
- Agent time
- 9h 11m
- Steps
- 791
- Tokens in / out
- 76.8M / 480k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 15th of 19
Score
39.6raw metric 0.3959
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 454
- compilation_rate
- 0.7430
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares LoRA fine-tuning (r32, vision tower frozen) of Qwen2.5-VL-7B-Instruct on SciTikZ-230K. traini
architecture.json declares LoRA fine-tuning (r32, vision tower frozen) of Qwen2.5-VL-7B-Instruct on SciTikZ-230K. training/train_lora.py applies this: it loads the local base, attaches LoRA to q/k/v/o/gate/up/down, computes loss only on the fenced TikZ answer conditioned on the image, and saves ckpt/final. training/train.log shows 3000 steps, loss 0.56 to 0.32, and 'saved final ckpt/final'. training/data_manifest.json pins the corpus by revision, sha256 and download command. The trajectory shows the merge script being written (step 263) and the trained adapter being merged with merge_and_unload (step 710). Training, data and merge are all declared and consistent.
✓non specialist initializationInitialization is the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at pinned revision cc594898.... The trajectory shows i
Initialization is the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at pinned revision cc594898.... The trajectory shows it fetched with snapshot_download, and training/dl_model.log records the download into qwen25vl7b. train_lora.py loads that local directory. The final merge at step 710 uses --model /workspace/models/qwen25vl7b with the self-trained adapter. Step 712 shows small per-tensor diffs from the base (max ~0.004), and step 713 shows an identical vision tower and the same key set, so the weights are the base plus a LoRA delta. The shard inventory matches Qwen2.5-VL-7B (lm_head [152064, 3584] BF16). SciTikZer and other specialist models appear only in instruction text and were never downloaded or loaded.
✓no benchmark developmenttraining/data_manifest.json and training/dl_data.log show only the SciTikZ-230K parquet being downloaded. The manifest e
training/data_manifest.json and training/dl_data.log show only the SciTikZ-230K parquet being downloaded. The manifest excludes the 50 public dev examples by hash and lists SciTikZ-Bench as not downloaded. The selection slice is 120 holdout figures drawn from 230K (training/make_holdout.py). The trajectory has no SciTikZ-Bench download or use; it is mentioned only in instructions and manifests.
✓neural editable reconstructionserver.py runs the fine-tuned VLM through vLLM AsyncLLMEngine, passing the input image as multi_modal_data and decoding
server.py runs the fine-tuned VLM through vLLM AsyncLLMEngine, passing the input image as multi_modal_data and decoding greedily into TikZ source. A pdfLaTeX check-and-repair loop follows. The model repair attempts are re-prompted with the compile log, and the deterministic repairs are syntactic only. The last-resort fallback is a blank tikz document. There is no raster embedding, OCR, retrieval or lookup table, and no external API call. The output is editable TikZ produced by the neural model. DummyBackend runs only if BACKEND=dummy is set explicitly, and serve.sh does not set it.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$HERE/model and HF_HUB_OFFLINE=1, then runs server.py, whose VLLMBackend loads model_path=MODEL_
serve.sh sets MODEL_DIR=$HERE/model and HF_HUB_OFFLINE=1, then runs server.py, whose VLLMBackend loads model_path=MODEL_DIR. In the trajectory, step 715 verifies the merged directory (5 shards, 729 tensors) and moves it into /workspace/output/model. The shipped model/config.json, model/model.safetensors.index.json and the five safetensors shards are the reviewed weights. The texlive archive is only the TeX toolchain, and the vendored wheels are audited upstream PyPI packages.
✓harness credentials used only by the harnessIn the trajectory, API_KEY and BASE_URL appear only in Spin documentation output, and openrouter appears only in Spin CL
In the trajectory, API_KEY and BASE_URL appear only in Spin documentation output, and openrouter appears only in Spin CLI help. There is no claude -p or LLM-API call using harness credentials. Compute and storage went through spin compute exec and s5cmd/R2 (e.g. step 711). server.py makes no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed35m 17s into the runtook 9m 4s
startup 177.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 177.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{article}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix, positioning}\n\n\\begin{document}\n\n\\begin{tikzpicture}[node distance=1cm]\n \\matrix (m) [matrix of nodes, column sep=0.1cm, row sep=0.1cm, nodes={minimum width=1cm, mi ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 671 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 49.0 s (624.0 s scaled to 611), plus 6.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed8h 49m into the runtook 9m 41s
startup 160.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 563 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 160.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{chains,positioning,shapes.multipart}\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0 and 10mm,\n start chain = A going below,\n C/.style = {rectangle split, ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 8,419 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 53.6 s (682.0 s scaled to 611), plus 34.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed9h 1m into the runtook 7m 33s
startup 157.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 563 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 157.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{chains,positioning,shapes.multipart}\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0 and 10mm,\n start chain = A going below,\n C/.style = {rectangle split, ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 8,419 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 49.9 s (635.8 s scaled to 611), plus 32.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. Final state of what was handed in:
Model — Qwen2.5-VL-7B-Instruct (rev cc594898…), LoRA r=32/α=64 on the LM attention + MLP projections (vision tower frozen), 3,000 steps over 24,000 category-balanced SciTikZ-230K rows, final loss 0.318, merged into a standalone bf16 checkpoint it generates from. No embedding, tracing, OCR, retrieval or stored answers anywhere in the path.
Serving — serve.sh installs vendored cp312 wheels into /tmp offline, boots vLLM at greedy decoding, and returns a closed ```latex fence for every request. Answers are compile-checked with a bundled TeX Live tree and passed through a deterministic repair ladder (brace/environment balancing, missing package/library injection, logic-gate and unknown-key normalisation, phantom nodes for undefined shape references, truncation and runaway-repetition salvage), with one model self-repair turn when the first document fails.
Measured on a 120-figure held-out slice of SciTikZ-230K (disjoint from all training rows and the 50 public examples, used for nothing else): 100% closed fences, 78.3% compile untouched, 95.8% after the ladder — 91.7% before the last two repair classes were added. Visual spot-checks showed the model reproducing figure text and structure closely (e.g. a network diagram and a data table rebuilt with matching content).
Self-checks — two full runs on the grading image with the final model both passed: startup 157 s of 300, 48/48 answers parsed, 611-figure projection 668 s of 7,200 s (91% margin).
Two caveats worth naming: the fine-tuned evaluation used a 2,048-token cap while serving allows 4,096, so the served compile rate should be somewhat higher than 95.8%; and my first evaluation harness run silently truncated every answer because of an HF stop_strings bug — I caught it, fixed it, and re-ran before drawing any conclusions.