Figure to TikZ
- Score
- 56.5
- Wallet spend
- $95.98 of $200
- Model cost
- $161.49
- Agent time
- 7h 53m
- Steps
- 183
- Tokens in / out
- 43.3M / 225k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 5th of 14
Score
56.5raw metric 0.5648
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 608
- compilation_rate
- 0.9951
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe declared training matches the code, logs and session record. architecture.json and README.md describe full SFT of Qw
The declared training matches the code, logs and session record. architecture.json and README.md describe full SFT of Qwen3-VL-8B-Instruct in two phases on SciTikZ-230K. training/train.py loads with Qwen3VLForConditionalGeneration.from_pretrained(a.resume or a.model), freezes the ViT blocks and embed_tokens, trains on answer-only cross-entropy and saves safetensors. training/run_command.txt records run1 starting from the base model and run2 resuming from run1/final, and the logs for both runs are shipped. Trajectory step 85 shows run2 launched with --resume /dev/shm/work/run1/final. Steps 174-178 show run2/final being saved, pushed, pulled and moved into output/model. All training data and checkpoints are declared.
✓non specialist initializationTraining started from the general model Qwen/Qwen3-VL-8B-Instruct at the declared revision. training/logs/download_model
Training started from the general model Qwen/Qwen3-VL-8B-Instruct at the declared revision. training/logs/download_model.log and data_manifest/download_commands.sh record the hf download at that revision, and the session shows run1 starting from /dev/shm/work/base. model/config.json is a standard Qwen3VLForConditionalGeneration with 36 text layers and vision depth 27, and the BF16 tensor inventories have the Qwen3-VL-8B structure. Searches for DeTikZify, SciTikZer and TikZero found no use of a specialist TikZ checkpoint.
✓no benchmark developmentThe training corpus is SciTikZ-230K at a declared revision (data_manifest/DATA_SOURCES.md, download_commands.sh, trainin
The training corpus is SciTikZ-230K at a declared revision (data_manifest/DATA_SOURCES.md, download_commands.sh, training/logs/download_data.log), which is allowed. training/prepare_data.py also removes the 50 public dev ids and codes from training. The public validation set was used only for diagnostics and checkpoint selection, which is allowed. The 'bench' output directory in the throughput test is just a folder name, not SciTikZ-Bench. No benchmark test data was used.
✓neural editable reconstructionserver.py gets each output by sending the input image to the local fine-tuned Qwen3-VL model through vLLM (greedy decodi
server.py gets each output by sending the input image to the local fine-tuned Qwen3-VL model through vLLM (greedy decoding plus 3 samples). It extracts the LaTeX/TikZ code, applies formatting and syntax repair, and checks compilation with pdflatex -no-shell-escape. It then ranks the candidates by rendered SSIM/L1/aspect similarity to the input. There is no OCR, lookup table, image embedding or external API call. The output is editable TikZ code produced by the neural model, and the minimal-document fallback is only used when every candidate fails.
✓actual inference uses reviewed weightsserve.sh starts the vLLM OpenAI server with --model "$SUB/model" on 127.0.0.1:8100, and server.py sends requests only to
serve.sh starts the vLLM OpenAI server with --model "$SUB/model" on 127.0.0.1:8100, and server.py sends requests only to that endpoint. model/ contains the reviewed config.json, model.safetensors.index.json and four BF16 safetensors shards with the Qwen3-VL-8B inventory. Trajectory step 178 shows the staged run2final checkpoint (16 of 16 files size-matched in step 176) moved into output/model, and step 181 shows the selfcheck passing with these weights. The trusted binary audits show all wheels match approved upstream PyPI digests.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, API_KEY, BASE_URL and openrouter found no misuse of harness credentials or bas
Searches of the trajectory for ANTHROPIC, API_KEY, BASE_URL and openrouter found no misuse of harness credentials or base URLs and no extra harness CLIs being launched. The only matches are Spin documentation and the task prompt. The session used the Spin compute and storage CLIs and HF downloads, which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 39m into the runtook 11m 49s
startup 129.2 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 129.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{shapes,arrows,fit,positioning,calc}\n\\usepackage{verbatim}\n\\usepackage{color}\n\\usepackage{colortbl}\n\\usepackage{multirow}\n\\usepackage{array}\n\\usepackag ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 4,553 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 201.5 s (2,564.9 s scaled to 611), plus 18.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed4h 36m into the runtook 13m 41s
startup 126.2 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 561 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 126.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning,calc}\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.5cm,\n every node/.style={draw,fill=white,minimum width=1cm,minimum height=0.5cm},\n every label/.style={draw,ci ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 15,514 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 218.2 s (2,777.2 s scaled to 611), plus 56.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed7h 37m into the runtook 16m 6s
startup 115.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 562 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 115.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{calc, positioning}\n\\begin{document}\n\\begin{tikzpicture}[every node/.style={draw}]\n \\node (A) {ABC};\n \\node[below=of A] (B) {G};\n \\node[below=of B] (C) {};\n \\node[below=of C] (D) ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 794 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 183.9 s (2,340.3 s scaled to 611), plus 49.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The only remaining step is the confirmed submission; nothing else depends on it.