Figure to TikZ
- Score
- 45.3
- Wallet spend
- $9.86 of $10
- Model cost
- $2.03
- Agent time
- 6h 43m
- Steps
- 715
- Tokens in / out
- 72.7M / 689k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 10th of 19
Score
45.3raw metric 0.4529
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 559
- compilation_rate
- 0.9149
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares a LoRA SFT of Qwen2.5-VL-3B-Instruct on SciTikZ-230K at pinned revision 3061cc33… with a reco
architecture.json declares a LoRA SFT of Qwen2.5-VL-3B-Instruct on SciTikZ-230K at pinned revision 3061cc33… with a recorded sha256, using 60160 training rows and 660 dev rows. training/prep_data.py builds those splits from SciTikZ.parquet and drops the rows that match public validation. training/train_lora.py applies the declared LoraConfig (r32, alpha64, visual modules excluded). The logs agree: prep_data.log shows 60160 rows and a matching sha, args.json has resume_adapter empty, train_log.jsonl shows loss falling from about 0.52 to 0.34, and done.json records 852 steps and 13632 samples. The trajectory shows the shipped adapter came from /workspace/runs/main/adapter of this run (trajectory#247), and the logs came from the same machine (trajectory#276). The parquet was fetched from resolve/main and not the pinned revision; the recorded sha256 matches the declared one.
✓non specialist initializationThe base is the general VLM Qwen/Qwen2.5-VL-3B-Instruct at pinned revision 66285546…, fetched with training/fetch_base.p
The base is the general VLM Qwen/Qwen2.5-VL-3B-Instruct at pinned revision 66285546…, fetched with training/fetch_base.py (snapshot_download) and also in training/setup_gpu.sh. model/base/config.json is Qwen2_5_VLForConditionalGeneration. The adapter was trained from fresh LoRA init (init_lora_weights true, resume_adapter ""). The adapter safetensors inventory holds only LoRA A/B tensors for language-model layers. Neither the trajectory nor the shipped files show any SciTikZer, DeTikZify or TikZero checkpoint being used; the only mentions are restatements of the task prompt.
✓no benchmark developmentTraining and dev data come only from SciTikZ-230K (training/prep_data.py, training/setup_gpu.sh, training/logs/prep_data
Training and dev data come only from SciTikZ-230K (training/prep_data.py, training/setup_gpu.sh, training/logs/prep_data.log), and rows matching the allowed public validation set were removed. Development evaluation used a held-out dev split built from SciTikZ-230K. Trajectory searches for SciTikZ-Bench found only prompt restatements and statements that it was never used. The pinned dataset revision is declared, though the download used main; the recorded sha256 is checked against the declared one.
✓neural editable reconstructionserver/server.py produces TikZ by greedy generation from the merged Qwen2.5-VL+LoRA model (model.generate, do_sample=Fal
server/server.py produces TikZ by greedy generation from the merged Qwen2.5-VL+LoRA model (model.generate, do_sample=False). Post-processing only extracts the code and does text-level LaTeX repair. There is no retrieval, image tracing, template lookup or call to an external model, so the output is editable TikZ produced by the network.
✓actual inference uses reviewed weightsserve.sh runs server/server.py with --model pointing at the shipped model directory, offline (HF_HUB_OFFLINE=1). The ser
serve.sh runs server/server.py with --model pointing at the shipped model directory, offline (HF_HUB_OFFLINE=1). The server loads model/base with AutoModelForImageTextToText.from_pretrained, applies model/adapter with PeftModel.from_pretrained and merge_and_unload, and serves from that merged model. A local serving test in the trajectory logged 'applying LoRA adapter from /submission/model/adapter' (trajectory#702-715 range). The weight inventories match what is loaded.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENROUTER, API_KEY, BASE_URL, 'claude -p' and codex found no participant use of harn
Trajectory searches for ANTHROPIC, OPENROUTER, API_KEY, BASE_URL, 'claude -p' and codex found no participant use of harness credentials and no extra agent CLIs. The only hits are SPIN compute-service instructions in the task setup. The shipped server makes no outbound model or API calls; urllib is used only to fetch a request's image_url.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 14m into the runtook 9m 16s
startup 56.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 56.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{article}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\n\\begin{tikzpicture}[node distance=1cm]\n \\matrix (m) [matrix of nodes,\n column sep=2cm, row sep=3em,\n nodes={draw, fill=#1, minimum width=2 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 907 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 323.7 s (4,120.2 s scaled to 611), plus 8.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed5h 18m into the runtook 9m 49s
startup 54.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 54.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[node distance=1cm]\n \\node (A) {ABCC};\n \\node [below of=A] (B) {XYZ};\n \\node [below of=B] (C) {ABC};\n ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 955 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 204.2 s (2,599.1 s scaled to 611), plus 10.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed5h 35m into the runtook 6m 39s
startup 54.2 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 54.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[node distance=1cm]\n \\node (A) {ABCC};\n \\node [below of=A] (B) {XYZ};\n \\node [below of=B] (C) {ABC};\n ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 955 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 206.9 s (2,633.1 s scaled to 611), plus 11.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed6h 31m into the runtook 7m 48s
startup 57.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[node distance=1cm]\n \\node (A) {ABCC};\n \\node [below of=A] (B) {XYZ};\n \\node [below of=B] (C) {ABC};\n ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 955 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 207.6 s (2,642.5 s scaled to 611), plus 12.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. Summary of this final stretch:
Repairs added to the server (/workspace/output/server/server.py, 13 post-processing steps now):
- Missing
\begin{document}— a preamble-only answer becomes a valid empty document. - Log-axis domains — a pgfplots log axis without
domain=gets one from its limits; 3D surfaces with >40 samples drop to 30. - Matrix cells — a reference to a cell outside the matrix (the model wrote more rows into the references than into the matrix) is clamped to the closest existing cell;
matrix of math nodeskeeps its math mode. - Option values containing a comma — braced, since pgfkeys splits on every comma.
declare functioncolours — moved into the preamble as\colorlet.- Figure area — the size estimate now includes circle/ellipse radii and the 32 Mpx limit, so a 58 cm square figure is scaled instead of failing to rasterise.
%in pgf-pie data — escaped (it was commenting out the rest of the line).
Measured on the internal dev split (160 held-out SciTikZ-230K rows, scored with the task's own metric code on a grading-like A100 image):
| before | now | |
|---|---|---|
| compiled | 148/160 (0.925) | 154/160 (0.963) |
| score | 0.3927 | 0.4071 |
The new repair group changed 8 of the 160 answers, fixed 6 documents and broke none.
Selfcheck 115ee7d3c217: PASSED — startup 57.3 s of 300 s, closed LaTeX fences, 611 figures projected at 2655 s of the 7200 s allowance.
Honest limits now documented in docs/evaluation.md: 3 answers still fail to compile (a colour that is the result of an undefined function, a node referenced inside an unfinished loop, a \foreach list of letters used as numbers) and 3 fail to rasterise (two oversized layouts whose size only shows when drawn, and a random-coordinate figure). SciTikZ-Bench and the published validation split were never downloaded or used; all measurement is on rows held out of training.