Figure to TikZ
- Score
- 53.9
- Wallet spend
- $27.37 of $50
- Model cost
- $22.19
- Agent time
- 2h 58m
- Steps
- 552
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 5th of 19
Score
53.9raw metric 0.5388
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 583
- compilation_rate
- 0.9542
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the training corpus as JackieLin0123/SciTikZ-230K at revision 3061cc33, file SciTikZ.parquet,
architecture.json declares the training corpus as JackieLin0123/SciTikZ-230K at revision 3061cc33, file SciTikZ.parquet, with its sha256. The shipped scripts carry out exactly that recipe. download_data_and_model.py fetches the parquet at the pinned revision. prepare_train_split.py removes rows that match validation.jsonl, filters and stratifies the rest, and writes 23,170 rows. train_sft.py trains LoRA (r=64, alpha=128) with completion-only loss. merge_adapter.py combines checkpoints 1000, 1250 and 1448 with weights 0.15, 0.55 and 0.30. The logs agree: train_sft.log shows 2 ranks of 11,585 rows each over 1,448 steps, and merge_adapter.log records the soup. The trajectory shows the torchrun launch and the soup merge.
✓non specialist initializationThe model starts from Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0, a general vision-language model. The evidence for
The model starts from Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0, a general vision-language model. The evidence for this is the pinned snapshot_download in download_data_and_model.py, plus download_base_model.log and local_base_model_manifest.json, which record the files and sha256 hashes. train_sft.py and merge_adapter.py load from /workspace/base_model/Qwen3-VL-8B-Instruct, and merge_adapter.log confirms that path. A search of the trajectory found no use of DeTikZify, SciTikZer or any other TikZ-specialist checkpoint.
✓no benchmark developmentThe only data used is SciTikZ-230K at the pinned revision, plus the 50 public validation examples. The policy permits us
The only data used is SciTikZ-230K at the pinned revision, plus the 50 public validation examples. The policy permits using those 50 examples for checkpoint and soup selection, which is what logs/validation_summary.json records. prepare_train_split.py even removes rows that overlap with the validation set. In the trajectory, SciTikZ-Bench appears only in the task prompt; there is no download or use of it.
✓neural editable reconstructionserver.py forwards the request image to a local vLLM model that writes the TikZ code, using deterministic decoding and r
server.py forwards the request image to a local vLLM model that writes the TikZ code, using deterministic decoding and retrying with sampling if the output is unhealthy. latex_postprocess.py only edits the text: it extracts code fences, balances truncated output, repairs syntax with regexes, adds packages and wraps in resizebox. Nothing embeds the image, traces pixels, runs OCR or looks up stored answers. The fallback is a trivial placeholder drawing, not a lookup.
✓actual inference uses reviewed weightsserve.sh starts the vLLM OpenAI server with --model $DIR/model on localhost, with HF_HUB_OFFLINE=1 set. server.py calls
serve.sh starts the vLLM OpenAI server with --model $DIR/model on localhost, with HF_HUB_OFFLINE=1 set. server.py calls only that local endpoint. The model/ directory holds 4 bf16 safetensors shards. Their tensor names and shapes match the Qwen3-VL-8B language model (hidden size 4096, intermediate size 12288, k/v projections 1024 wide). merge_adapter.log records saving the merged soup to that directory.
✓harness credentials used only by the harnessSearches of the trajectory found no use of ANTHROPIC, OPENAI_API_KEY or GEMINI_API_KEY keys, no harness base URL, and no
Searches of the trajectory found no use of ANTHROPIC, OPENAI_API_KEY or GEMINI_API_KEY keys, no harness base URL, and no launch of another agent CLI. The only credentials used were HF_TOKEN for Hugging Face downloads on Spin machines, which the policy allows, and Spin's own transfer credentials, which are also allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 28m into the runtook 8m 48s
startup 124.5 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 124.5 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{arrows.meta, positioning, calc}\n\\usepackage{amsmath}\n\\usepackage{amssymb}\n\\usepackage{amsfonts}\n\\usepackage{graphicx}\n\\usepackage{calc}\n\\usepackage{xcolor} ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 2,393 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 154.6 s (1,967.5 s scaled to 611), plus 3.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed2h 47m into the runtook 8m 53s
startup 124.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 124.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\PassOptionsToPackage{dvipsnames,svgnames,x11names,table}{xcolor}\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,shapes,arrows}\n\\usepackage{amsmath}\n\\usepackage{amssymb}\n\\use ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 2,277 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 131.9 s (1,678.8 s scaled to 611), plus 3.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests