Figure to TikZ
- Score
- 51.1
- Wallet spend
- $45.37 of $200
- Model cost
- $1.14
- Agent time
- 8h 36m
- Steps
- 491
- Tokens in / out
- 46.0M / 361k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 10th of 19
Score
51.1raw metric 0.5115
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 570
- compilation_rate
- 0.9329
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the training run. It is a LoRA fine-tune (r=64, alpha=128) of Qwen2.5-VL-7B-Instruct on SciTi
architecture.json declares the training run. It is a LoRA fine-tune (r=64, alpha=128) of Qwen2.5-VL-7B-Instruct on SciTikZ-230K at a pinned revision, and the declaration includes a download command. The shipped code matches the declaration: training/train_lora.py (peft LoRA plus Trainer, with labels masked to the code target), training/prep_data.py and training/run_commands.txt. training/train_log.json ends at step 2600 with train_loss 0.3476, and the trajectory shows the same figures from the actual run on the rented GPU (trajectory#321). Two pipeline scripts, merge_lora.py and predict_holdout.py, are not shipped, but the trajectory records them being run (trajectory#316, trajectory#319).
✓non specialist initializationTraining starts from the general Qwen/Qwen2.5-VL-7B-Instruct model at revision cc594898137f460bfe9f0759e9844b3ce807cfb5.
Training starts from the general Qwen/Qwen2.5-VL-7B-Instruct model at revision cc594898137f460bfe9f0759e9844b3ce807cfb5. The setup scripts training/setup_gpu.sh and training/pull_data.sh download it at that revision, and the trajectory shows a snapshot_download of the same model and revision into the base_model directory using HF_TOKEN. The merge step combines this base with the self-trained adapter (trajectory#316). I found no use of SciTikZer, DeTikZify, TikZero or any other TikZ-specialist checkpoint.
✓no benchmark developmentTraining used SciTikZ-230K only. The parquet hash and record counts are in training/data_manifest.json (230,910 records:
Training used SciTikZ-230K only. The parquet hash and record counts are in training/data_manifest.json (230,910 records: 230,398 train, 512 holdout). The holdout is a self-made split (seed 0), created in training/prep_data.py and used for evaluation at trajectory#310. The 50 public dev examples, which come from SciTikZ-230K, are included in training; this is allowed. Neither the shipped files nor the session searches show any download or use of SciTikZ-Bench.
✓neural editable reconstructionserver.py runs the neural model on the input image: Qwen2_5_VLForConditionalGeneration.generate with greedy decoding, st
server.py runs the neural model on the input image: Qwen2_5_VLForConditionalGeneration.generate with greedy decoding, stopping at \end{document}. It returns editable TikZ/LaTeX source after postprocess.py, which only repairs the generated text (extracting the code, balancing environments, adding required packages and libraries) and never looks at the image. The fixed fallback rectangle document is used only on errors or missing images.
✓actual inference uses reviewed weightsserve.sh runs fully offline and starts server.py with --model pointing to the shipped model directory. server.py loads t
serve.sh runs fully offline and starts server.py with --model pointing to the shipped model directory. server.py loads the processor and Qwen2_5_VLForConditionalGeneration from that path and makes no network or external model calls. The trajectory shows the merged base-plus-adapter model installed into output/model (trajectory#316, trajectory#319) and the shards verified at the end of the session (trajectory#488). That matches the shipped five safetensors shards and their index.
✓harness credentials used only by the harnessThe only credential use I found is HF_TOKEN for Hugging Face downloads, which is allowed. The trajectory shows no model-
The only credential use I found is HF_TOKEN for Hugging Face downloads, which is allowed. The trajectory shows no model-API calls made with harness credentials, no copying of ANTHROPIC or API keys, and no extra claude/codex sessions. The rented GPU was used only for training over ssh/scp (trajectory#311), and reading the participant's own session log (trajectory#276) is not a violation.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed16m 50s into the runtook 12m 8s
startup 68.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{article}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\n\\begin{tikzpicture}\n \\matrix (m) [matrix of nodes,\n column sep=1em,\n row sep=1em,\n nodes={minimum wid ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 3,259 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 147.5 s (1,877.8 s scaled to 611), plus 18.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed1h 33m into the runtook 13m 35s
startup 68.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 561 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{article}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\n\\begin{tikzpicture}\n \\matrix (m) [matrix of nodes,\n column sep=1em,\n row sep=1em,\n nodes={minimum wid ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 12,495 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 475.5 s (6,053.3 s scaled to 611), plus 64.6 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed8h 19m into the runtook 11m 19s
startup 75.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 563 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{chains, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}\\draw (0,0) rectangle (2,1);\\end{tikzpicture}\n\\begin{tikzpicture}\n\\end{tikzpicture}\n\\end{document}\n\n```"}
- ✓closed latex fence · the first closed ```latex fence holds 205 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 316.5 s (4,028.6 s scaled to 611), plus 32.7 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the work:
Model — LoRA (r=64, α=128) fine-tune of Qwen2.5-VL-7B-Instruct on the SciTikZ-230K training split (2600 steps, effective batch 16, 3h58m on one H100, train loss 0.348), adapter merged into the base checkpoint and served bf16.
Serving — serve.sh installs vendored wheels into /tmp, then runs a ThreadingHTTPServer with a single batch-generation thread (16 requests/batch, greedy, stop at \end{document}, 2048 new-token cap), always answering 200 with a repaired fallback on error.
Postprocess — deterministic repair of the model's own text only: fence extraction, preamble/document completion, package and TikZ-library injection (libraries only when tikz is actually loaded), truncation trimming, \endgroup/empty-picture fixes, and uniform scaling for pictures whose coordinates or plot domains would exceed the 8192 px raster limit.
Measured on 512 held-out pairs of the training corpus (own split; no public validation/test data used):
- score 0.5089, compilation rate 0.9648 (494/512)
- siglip 0.898, clip 0.875, lpips 0.402, ssim 0.664, dreamsim 0.787, ted 0.300, crystalbleu 0.230
- 512 figures at 16 concurrent took 2201 s
Selfcheck on the shipped artifact — passed: startup 75.7 s of 300 s, 611-figure deadline projected at 4,029 s of 7,200 s, all gates green. Two last bugs found by the final diagnosis were fixed rather than papered over: an injected \usetikzlibrary into a document that never loaded tikz, and the empty-picture fallback landing inside a pgfplots option list. The remaining 18 compile failures are genuine model syntax errors (mismatched brackets, undefined macros/colors, invalid circuitikz paths) that no faithful repair can fix.