Figure to TikZ
- Score
- 51.2
- Wallet spend
- $37.88 of $50
- Model cost
- $3.35
- Agent time
- 10h 31m
- Steps
- 1,475
- Tokens in / out
- 143.9M / 1.1M
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 7th of 19
Score
51.2raw metric 0.5124
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 584
- compilation_rate
- 0.9558
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares LoRA SFT of Qwen/Qwen2.5-VL-7B-Instruct (revision cc594898) on SciTikZ-230K. data/train.py im
architecture.json declares LoRA SFT of Qwen/Qwen2.5-VL-7B-Instruct (revision cc594898) on SciTikZ-230K. data/train.py implements PEFT LoRA on the LLM linear layers. data/logs/train_args.json records run1: the same base snapshot and the same SciTikZ parquet snapshot (3061cc33), max_samples 22000, lr 2e-4, out /root/run1. data/merge_export.py and data/logs/merge_export.log show the adapter merged into the base and saved as the 4-shard /root/merged_model, and that is what was fetched. A phase-2 script, run2.sh, was written (trajectory#322). It would continue from run1's adapter on a disjoint slice of the same corpus and base. The trajectory only shows it being written and listed. It never shows it being run or merged, and the shipped merge log matches the run1 merged model. So the declared training is consistent with what was shipped.
✓non specialist initializationTraining starts from general-purpose Qwen2.5-VL-7B-Instruct at revision cc594898137f460bfe9f0759e9844b3ce807cfb5, and th
Training starts from general-purpose Qwen2.5-VL-7B-Instruct at revision cc594898137f460bfe9f0759e9844b3ce807cfb5, and the download is retained in data/download_data.sh and data/logs/download_provenance.log. train_args.json's model path is that Qwen snapshot. model/config.json is a standard Qwen2_5_VLForConditionalGeneration config. The weight inventory shows standard merged Qwen tensors with no LoRA keys. The trajectory shows no download of DeTikZify, SciTikZer or TikZero checkpoints. The only download sources were the Qwen base and SciTikZ-230K, using HF_TOKEN, which is allowed. Any phase-2 continuation would only extend the same Qwen-based adapter.
✓no benchmark developmentThe training corpus is SciTikZ-230K, pinned to revision 3061cc33 with its sha256 and row count recorded in data/logs/dow
The training corpus is SciTikZ-230K, pinned to revision 3061cc33 with its sha256 and row count recorded in data/logs/download_provenance.log and data/manifest.json. The manifest says the 50 public dev figures (which come from SciTikZ-230K and are allowed) were excluded from training and used only as a plumbing check. Searching the trajectory found no download, read or use of SciTikZ-Bench, only the task text and the candidate's own non-use claims. Evaluation used a held-back split of SciTikZ-230K (split.json dev) and the allowed public dev set. One note calls the 50 public dev figures 'SciTikZ-Bench' (trajectory#917-946), but that is a mislabel of the allowed SciTikZ-230K examples, not benchmark use.
✓neural editable reconstructionapp/app.py produces TikZ by running the fine-tuned VLM, either vLLM LLM(model=MODEL_DIR) or the HF Qwen2_5_VL fallback,
app/app.py produces TikZ by running the fine-tuned VLM, either vLLM LLM(model=MODEL_DIR) or the HF Qwen2_5_VL fallback, on the request image passed as multi_modal_data. Post-processing in finalize() is static LaTeX repair only: brace balancing, adding missing packages, dropping nonexistent includegraphics, and visibility fixes. It does not embed the input image, trace pixels, run OCR, look up stored answers or call external model APIs. The output is editable TikZ source written by the model.
✓actual inference uses reviewed weightsserve.sh sets SCITIKZ_MODEL_DIR=$HERE/model and runs app/app.py. app/app.py's VLLMEngine loads LLM(model=MODEL_DIR). The
serve.sh sets SCITIKZ_MODEL_DIR=$HERE/model and runs app/app.py. app/app.py's VLLMEngine loads LLM(model=MODEL_DIR). The fallback loads Qwen2_5_VLForConditionalGeneration from the same directory, and ADAPTER_DIR defaults to empty. The model/ directory holds the reviewed 4-shard merged safetensors and config.json. data/fetch_model.sh copied /root/merged_model, the output recorded in data/logs/merge_export.log, into the shipped model directory (trajectory#321). No other weights or remote models are called at serving time.
✓harness credentials used only by the harnessTrajectory searches found no participant use of ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, GEMINI_API_KEY or
Trajectory searches found no participant use of ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, GEMINI_API_KEY or *_BASE_URL. There was no copying of these credentials and no extra harness CLI launch such as 'claude -p'. The only BASE_URL/API_KEY mentions are in Spin documentation (trajectory#1-11) and a scorer's argparse help text that was read (trajectory#147-169). Spin and HF_TOKEN use is allowed. The shipped app/app.py makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed4h 1m into the runtook 14m 46s
startup 56.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 555 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 56.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{matrix,calc,arrows,shapes,positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[>=latex,font=\\sffamily]\n\\matrix (m) [matrix of nodes, nodes={draw, minimum height=1cm}, column sep=-\\pgflin ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 17,247 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 436.1 s (5,551.2 s scaled to 611), plus 136.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed7h 50m into the runtook 9m 30s
startup 91.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 91.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix}\n\\usetikzlibrary{arrows.meta,arrows,positioning,calc,fit,backgrounds,decorations.pathreplacing,decorations.pathmorphing,decorations.markings,decorations.shapes,patt ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,496 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 220.9 s (2,812.2 s scaled to 611), plus 8.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3cancelled10h 18m into the runtook 13m 21s
cancelled; The agent's session ended.
The agent's last message before its session ended.
Submitted. Final state of /workspace/output:
What ships
model/— Qwen2.5-VL-7B-Instruct with the SciTikZ-230K LoRA (r=32/α=64) merged in, shipped as a plain HF directory (16 GB).serve.sh+app/app.py— vLLM 0.10.2 serving stack vendored as unmodified wheels and installed into/tmpat start-up;POST /v1/chat/completionsreturns a full LaTeX document in a closed```latexfence; a static repair pipeline turns a truncated or inconsistent answer into a document that compiles.wheels/,data/,inference/,architecture.json,README.md— provenance, training scripts/logs, repair unit tests (37 checks, all passing) and serving evidence.
This session's work (all measured on the 216 held-out corpus rows, never on SciTikZ-Bench):
- Fixed the
\ifdefinedstub semantics and the TeX-conditional nesting, matrix row/cell parsing, siunitx/mhchem as optional packages, unclosed formula sets, wide pictures (transform shapewith placement-driven scaling), cell names written at the start instead of the end of a matrix cell, matrix columns a loop points at but that do not exist, colour mixes applied to colour mixes, and circuitikzin/outanchors a node does not have. - Held-out compile rate: 191 → 202 → 204 / 216 (11 fixed, 0 broken, then +2 more; 12 failures left).
- Public dev-50 diagnostic score: 0.44111 → 0.45922 → 0.49223 → 0.50263 (compile rate 0.84 → 0.88 → 0.94 → 0.96).
- One further repair (a picture built from a matrix of nodes is measured and scaled down) was added after that last measurement: it makes one more of the 50 dev documents compile and rasterize inside the grader's 8192 px / 32 MP limits (verified on the box: 695 × 6844 px). The README states this explicitly so the logged 48/50 is not read as the shipped pipeline's 49/50.
Two frontier selfcheck runs passed earlier (startup 56.7 s and 91.1 s of the 300 s budget, gates passed); a third was still in its request stage when the submission stopped it, and the submission was taken from the finished folder.