Figure to TikZ
- Score
- 40.7
- Wallet spend
- $2.93 of $10
- Model cost
- $0.78
- Agent time
- 6h 1m
- Steps
- 326
- Tokens in / out
- 30.5M / 254k
- Self-checks
- 2 passed of 4 used
- Rank at $10 wallet
- 15th of 19
Score
40.7raw metric 0.4073
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 482
- compilation_rate
- 0.7889
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares a LoRA SFT (r=32, alpha=64) of Qwen2.5-VL-3B-Instruct on 26,433 rows of SciTikZ-230K, then a
architecture.json declares a LoRA SFT (r=32, alpha=64) of Qwen2.5-VL-3B-Instruct on 26,433 rows of SciTikZ-230K, then a bf16 merge. The shipped code matches this declaration. train/train_lora.py applies peft LoRA to q/k/v/o/gate/up/down, keeps the vision tower frozen and computes loss only on assistant tokens. train/run_train.sh gives the same hyperparameters. train/train_log.jsonl records the start args (rows 26433, lr 1.5e-4, lora_r 32) and the per-step losses. eval/merge_adapter.py merges the adapter with merge_and_unload. In the trajectory, the merged model was downloaded from Spin S3 (step 244) and moved into output/model, and train_log.jsonl was fetched from the run's output directory (step 253).
✓non specialist initializationTraining starts from the general VLM Qwen/Qwen2.5-VL-3B-Instruct at pinned revision 66285546. train/download_model.py ru
Training starts from the general VLM Qwen/Qwen2.5-VL-3B-Instruct at pinned revision 66285546. train/download_model.py runs snapshot_download at that revision, and download_model.log shows the fetch from huggingface.co succeeding. train_log.jsonl shows the base is /root/model, the uploaded Qwen snapshot. The weight inventory has Qwen2.5-3B shapes (embed 151936x2048, MLP 11008) and no foreign or specialist structure. I found no TikZ-specialist checkpoint (e.g. DeTikZify or SciTikZer) loaded anywhere.
✓no benchmark developmentThe training corpus is SciTikZ-230K, which is allowed: train/download_dataset.py fetches it at the declared revision. tr
The training corpus is SciTikZ-230K, which is allowed: train/download_dataset.py fetches it at the declared revision. train/select_data.py removes rows whose code matches public/validation.jsonl, deduplicates the rest and splits off a 348-row holdout from the same corpus. data_manifest.json records the counts. Searching the code and the trajectory found no download or use of SciTikZ-Bench. Using the 50 public validation examples for diagnostics is allowed.
✓neural editable reconstructionFor each request, server.py decodes the base64 image and runs Qwen2_5_VLForConditionalGeneration.generate (greedy, with
For each request, server.py decodes the base64 image and runs Qwen2_5_VLForConditionalGeneration.generate (greedy, with a no-repeat-ngram logits processor) to produce TikZ code. postproc.py only makes deterministic text repairs (document selection, xcolor fixes, package insertion, closing truncated environments). It does not retrieve or synthesize figure content, and there are no lookup tables or raster embedding.
✓actual inference uses reviewed weightsserve.sh sets offline mode and execs server.py. server.py loads MODEL_DIR=HERE/model with from_pretrained, and that dire
serve.sh sets offline mode and execs server.py. server.py loads MODEL_DIR=HERE/model with from_pretrained, and that directory contains the inventoried shards model-00001/00002-of-00002.safetensors. The trajectory shows output/model was replaced by the merged checkpoint downloaded from S3 (steps 244 and 253). Later tests at steps 285-326 load /workspace/output/model directly. There is no remote API or alternative model path.
✓harness credentials used only by the harnessThe trajectory shows only allowed credential use. HF_TOKEN was used for Hugging Face downloads, and Spin CLI compute and
The trajectory shows only allowed credential use. HF_TOKEN was used for Hugging Face downloads, and Spin CLI compute and transfer credentials were used for the node and S3 (around steps 66-67, 244 and 248-253). Searches for API_KEY and BASE_URL found harness credentials only in the task instructions. No participant code or command used ANTHROPIC/OPENAI keys or base URLs, and no extra harness CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 0m into the runtook 16m 16s
startup 68.8 s of 300 s; 0/1 gates passed; failed 611 figures in 7200 s: projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 558.3 s share of the deadline after a 93.5 s first figure (Batch exceeded 558.288s deadline; score is zer
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{article}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\n\\begin{tikzpicture}[node distance=1cm]\n \\matrix (m) [matrix of nodes,\n column sep=2cm, row sep=3em,\n nodes={draw, fill=#1, minimum width=2 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 12,395 characters of LaTeX
- ✗611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 558.3 s share of the deadline after a 93.5 s first figure (Batch exceeded 558.288s deadline; score is zero)
- 2failed4h 25m into the runtook 13m 35s
startup 70.5 s of 300 s; 0/1 gates passed; failed 611 figures in 7200 s: projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 564.8 s share of the deadline after a 10.2 s first figure (Batch exceeded 564.828s deadline; score is zer
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.5 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0mm,\n box/.style = {draw, minimum width=1cm, minimum height=1cm},\n circ/.style = {circle, draw, font=\\bfse ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 704 characters of LaTeX
- ✗611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 564.8 s share of the deadline after a 10.2 s first figure (Batch exceeded 564.828s deadline; score is zero)
- 3passed5h 5m into the runtook 13m 41s
startup 88.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 88.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0mm,\n box/.style = {draw, minimum width=1cm, minimum height=1cm},\n circ/.style = {circle, draw, font=\\bfse ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 704 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 396.9 s (5,052.6 s scaled to 611), plus 7.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed5h 50m into the runtook 7m 23s
startup 84.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0mm,\n box/.style = {draw, minimum width=1cm, minimum height=1cm},\n circ/.style = {circle, draw, font=\\bfse ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 704 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 183.6 s (2,336.5 s scaled to 611), plus 7.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output/:
Model — Qwen2.5-VL-3B-Instruct (rev 66285546…) fine-tuned with LoRA r=32 on 26,433 rows of SciTikZ-230K's train split (dev-50 codes excluded by hash), merged to bf16 safetensors. SciTikZ-Bench was never touched; provenance is in architecture.json plus the download scripts and logs under train/.
Serving — serve.sh installs 42 vendored cp312 wheels into /tmp (~60 s), then runs a stdlib HTTP server with:
- chunked decoding (512-token rounds, finished sequences free their slot for queued requests) — worth 2–3× over static batching, since a single 3k-token figure otherwise stalls a whole batch of 16;
- an incremental no-repeat-n-gram logits processor: identical bans to the HF stock processor (verified token-identical on CPU), but O(1) per step instead of rescanning the prefix, which was costing ~90 ms/step at 3k tokens;
- deterministic text repairs (xcolor option clash, missing tikz/pgfplots, truncated-stream closing) that fixed the corpus's own 2.6% un-compilable reference documents.
Measured results
- Final selfcheck: PASSED all gates — startup 84.1 s of 300 s, 611 figures projected at 2,343.85 s of 7,200 s (67% margin), all 48 answers parsed with closed ```latex fences.
- Held-out 96-figure subset (348-row internal holdout, never trained on): 73/96 = 76.0% compile, SigLIP 0.869, CLIP 0.835, SSIM 0.705, DreamSim 0.725, TED 0.232, CrystalBLEU 0.096 → score ≈ 0.39.