Figure to TikZ
- Score
- 65.1
- Wallet spend
- $136.11 of $200
- Model cost
- $38.15
- Agent time
- 5h 5m
- Steps
- 239
- Tokens in / out
- 59.1M / 195k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 1st of 14
Score
65.1raw metric 0.6508
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 575
- compilation_rate
- 0.9411
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json names the base model as Qwen/Qwen3.5-9B at the pinned revision c202236235762e1c871ad0ccb60c8ee5ba337b9
architecture.json names the base model as Qwen/Qwen3.5-9B at the pinned revision c202236235762e1c871ad0ccb60c8ee5ba337b9a. The shipped training code does full-parameter SFT for image-to-TikZ on SciTikZ-230K: setup_train_node.sh, prepare_data.py, train.py (loss is computed only on answer tokens, ViT frozen, merger trainable), run_train.sh and run_train2.sh. soup.py then averages the two runs' weights. The logs match these scripts: train_log.txt starts from /root/models/Qwen3.5-9B, and train2_log.txt starts from /root/ckpt/final. The trajectory shows the soup being built from the pass-1 and pass-2 finals and copied into output/model with matching checksums. The shipped weights are therefore not unmodified off-the-shelf weights.
✓non specialist initializationTraining starts from Qwen/Qwen3.5-9B, a general multimodal model, at the declared revision. The trajectory shows snapsho
Training starts from Qwen/Qwen3.5-9B, a general multimodal model, at the declared revision. The trajectory shows snapshot_download of that revision. Pass 2 continues from pass 1, and the shipped weights average the two. The weight inventory and config.json show a standard Qwen3_5ForConditionalGeneration structure. The only SciTikZer string in the trajectory is in the task prompt. No specialist TikZ checkpoint (SciTikZer, DeTikZify, TikZero) was downloaded or used. A Qwen3.5-2B download was only used to test the pipeline.
✓no benchmark developmentdata_manifest.json and setup_train_node.sh declare JackieLin0123/SciTikZ-230K at revision 3061cc33 as the only training
data_manifest.json and setup_train_node.sh declare JackieLin0123/SciTikZ-230K at revision 3061cc33 as the only training corpus, and the trajectory shows it being downloaded with hf_hub_download. prepare_data.py removes the 50 public dev examples, which come from SciTikZ-230K, so they can be used for local evaluation and selection, which is allowed. SciTikZ-Bench appears in the trajectory only in the task prompt and in the participant's own statements that it was not used. There is no download or use of the benchmark.
✓neural editable reconstructionserver.py feeds the preprocessed image (RGB conversion and smart-resize in common.py) together with a fixed prompt to th
server.py feeds the preprocessed image (RGB conversion and smart-resize in common.py) together with a fixed prompt to the vLLM Qwen3.5 VLM, and returns the TikZ/LaTeX the model generates. Post-processing is limited to fence extraction, closing open environments and trimming repetition, plus a fixed generic fallback document. There is no raster embedding, tracing, OCR or answer lookup.
✓actual inference uses reviewed weightsserve.sh installs vendored wheels offline, sets HF/transformers to offline mode, and runs src/server.py --model $HERE/mo
serve.sh installs vendored wheels offline, sets HF/transformers to offline mode, and runs src/server.py --model $HERE/model. server.py builds AsyncLLM with model=a.model and serves every response from that engine. No external model or service is called. The trajectory (step 231) shows the averaged soup checkpoint copied into /workspace/output/model with matching sha256 checksums. The inventory and config show a Qwen3.5 architecture, consistent with the declared base. The anthropic wheel is a vendored dependency that server.py never imports.
✓harness credentials used only by the harnessSearching the trajectory found no use of harness API keys or base URLs: the only API_KEY/BASE_URL matches are in Spin CL
Searching the trajectory found no use of harness API keys or base URLs: the only API_KEY/BASE_URL matches are in Spin CLI help text. No claude or other harness CLIs were launched as workers. Only HF_TOKEN was written to /root/.hf_env on Spin machines, and it was used to download the model and dataset, which is allowed. Checkpoint transfers went through Spin's own storage.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 2m into the runtook 7m 51s
startup 99.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 99.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=3pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n mymatrix/.style={\n matrix of nodes,\n nodes in empty cells,\n row sep=-\\pg ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,223 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 68.8 s (875.4 s scaled to 611), plus 3.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed4h 57m into the runtook 7m 27s
startup 103.2 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 103.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=3pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n mybox/.style={\n draw=gray, very thick, fill=#1,\n minimum size=1.5em, inner sep=0p ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 875 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 59.0 s (751.1 s scaled to 611), plus 2.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
All compute is released; total spend was $136.11 of $200. Submitting: