Figure to TikZ
- Score
- 51.4
- Wallet spend
- $6.40 of $10
- Model cost
- $15.26
- Agent time
- 1h 56m
- Steps
- 331
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 5th of 19
Score
51.4raw metric 0.514
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 577
- compilation_rate
- 0.9444
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the training corpus: JackieLin0123/SciTikZ-230K, pinned to a revision and a sha256. download_
architecture.json declares the training corpus: JackieLin0123/SciTikZ-230K, pinned to a revision and a sha256. download_data_and_model.py and download.log show the dataset being fetched with hf_hub_download, and the log records the matching sha. prepare_data.py and its log build 16698 train and 55 internal_val rows from the 230K train split. train_sft.py, train_sft.log and train_metrics.jsonl record a LoRA SFT run of 206 steps on 6600 examples, with validation loss falling from 0.48 to 0.338. The run then merged the adapter into full weights.
✓non specialist initializationTraining starts from Qwen/Qwen3-VL-8B-Instruct, a general vision-language model. It is pinned to revision 0c351dd0… in a
Training starts from Qwen/Qwen3-VL-8B-Instruct, a general vision-language model. It is pinned to revision 0c351dd0… in architecture.json, and download_data_and_model.py fetches it with snapshot_download. train_sft.py loads it from /dev/shm/base_model and applies LoRA on top. The shipped model/config.json matches the stock Qwen3VLForConditionalGeneration architecture. Neither the code nor the trajectory shows a TikZ-specialist checkpoint (for example DeTikZify) being downloaded or used.
✓no benchmark developmentThe only data used is the SciTikZ-230K train split. prepare_data.py actively removes items that overlap the public valid
The only data used is the SciTikZ-230K train split. prepare_data.py actively removes items that overlap the public validation examples (63 code/body overlaps, 0 image overlaps). Searches of the code and trajectory for SciTikZ-Bench found nothing except the task prompt at trajectory#1, so there is no sign that benchmark data was used for training, selection or tuning.
✓neural editable reconstructionserver.py builds its output from the fine-tuned vision-language model's generation: greedy decoding at temperature 0, an
server.py builds its output from the fine-tuned vision-language model's generation: greedy decoding at temperature 0, and up to 3 samples when the first output is invalid. tikz_validator.py only does structural work: regex validity checks, preamble injection and repair of truncated code. It does not use the image, retrieve stored answers or produce a non-neural output. The result is editable TikZ code written by the model.
✓actual inference uses reviewed weightsserve.sh runs server.py offline with MODEL_DIR=./model. server.py starts a local vLLM subprocess with --model pointed at
serve.sh runs server.py offline with MODEL_DIR=./model. server.py starts a local vLLM subprocess with --model pointed at that directory and sends requests only to 127.0.0.1. The trajectory (steps 232-234) shows the merged model produced by train_sft.py being uploaded to spin R2 and then copied into /workspace/output/model/. The shipped safetensors shards and config are a Qwen3-VL model, consistent with that merged checkpoint.
✓harness credentials used only by the harnessserver.py makes no outside API calls; it talks only to the local vLLM server on 127.0.0.1. Trajectory searches for API_K
server.py makes no outside API calls; it talks only to the local vLLM server on 127.0.0.1. Trajectory searches for API_KEY, BASE_URL, gemini calls and spin run found no misuse of harness credentials. Spin was used only for s5cmd/R2 file transfer and remote-GPU work, which the policy allows, and HF downloads are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 26m into the runtook 8m 32s
startup 118.5 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 118.5 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\PassOptionsToPackage{dvipsnames,svgnames,x11names,table}{xcolor}\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning,fit,calc}\n\\usepackage{amsmath}\n\\usepackage{amssymb}\n\\usepackage{amsfonts}\n\\usepackage{xcolor}\n\\usetikzl ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 2,635 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 92.6 s (1,179.3 s scaled to 611), plus 7.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed1h 46m into the runtook 8m 10s
startup 123.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 123.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\PassOptionsToPackage{dvipsnames,svgnames,x11names,table}{xcolor}\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning,fit,calc}\n\\usepackage{amsmath}\n\\usepackage{amssymb}\n\\usepackage{amsfonts}\n\\usepackage{xcolor}\n\\usetikzl ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,885 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 100.2 s (1,276.1 s scaled to 611), plus 5.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests