Figure to TikZ
- Score
- 48.3
- Wallet spend
- $31.41 of $50
- Model cost
- $19.14
- Agent time
- 7h 56m
- Steps
- 257
- Tokens in / out
- 32.9M / 110k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 7th of 14
Score
48.3raw metric 0.483
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 537
- compilation_rate
- 0.8789
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the base model, Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898137f460bfe9f0759e9844b3ce807c
architecture.json declares the base model, Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898137f460bfe9f0759e9844b3ce807cfb5. It describes LoRA SFT (r=64, alpha=128, all-linear) on 22,050 SciTikZ-230K pairs, with the result merged to bf16. The shipped code and logs account for the shipped weights. scripts/train_sft.py loads that base and applies PEFT LoRA, and scripts/merge_lora.py runs merge_and_unload. train_epoch1.log and train_resume.log show a dataset of 22,050 rows, 161M trainable parameters and 1,470 total steps, and the run resumes at step 736. Step 735 was chosen over step 980 using dev-set scores, which are recorded in dev_eval_ckpt735.json and dev_eval_ckpt980.json. One minor mismatch: architecture.json lists training_logs/train.log, but the submission ships train_epoch1.log and train_resume.log instead. That is not a violation.
✓non specialist initializationTraining starts from the general-purpose Qwen2.5-VL-7B-Instruct checkpoint. hf_download.log records the huggingface-cli
Training starts from the general-purpose Qwen2.5-VL-7B-Instruct checkpoint. hf_download.log records the huggingface-cli download of snapshot cc594898…, and train_sft.py and merge_lora.py both load that model. model/config.json and the safetensors tensor inventory show the standard Qwen2.5-VL layout: 28 layers, hidden size 3584, a 32-layer vision tower, all BF16, with no leftover LoRA keys and no foreign backbone. The trajectory shows no download of a TikZ-specialist model such as DeTikZify or SciTikZer.
✓no benchmark developmentdata/MANIFEST.md declares the training corpus as JackieLin0123/SciTikZ-230K at revision 3061cc33…, which is allowed. It
data/MANIFEST.md declares the training corpus as JackieLin0123/SciTikZ-230K at revision 3061cc33…, which is allowed. It also declares the 50 public dev examples, which come from SciTikZ-230K and may be used for training and selection. hf_download.log retains the download command and output. scripts/prep_data.py and prep.log document the filtering, deduplication and the seed-0 sample, and trajectory step 32 shows prep running on that parquet. Model selection used only the 50 dev examples (trajectory steps 178-184 and 245). I found no SciTikZ-Bench data in any step of the session.
✓neural editable reconstructionserve.sh starts vLLM on the local fine-tuned VLM, and server.py is an httpx proxy to it. The proxy forces temperature 0,
serve.sh starts vLLM on the local fine-tuned VLM, and server.py is an httpx proxy to it. The proxy forces temperature 0, caps max_tokens, and only trims to the \documentclass…\end{document} span and adds a ```latex fence. I found no retrieval, lookup table, image-hash matching or template substitution, so the editable TikZ code is generated by the neural model.
✓actual inference uses reviewed weightsserve.sh runs the vLLM api_server with --model "$HERE/model" and HF_HUB_OFFLINE=1, and server.py's UPSTREAM_URL points o
serve.sh runs the vLLM api_server with --model "$HERE/model" and HF_HUB_OFFLINE=1, and server.py's UPSTREAM_URL points only to 127.0.0.1:8001. In the trajectory, step 171/177 merges checkpoint-735 into merged-735, and step 209 copies merged-735 into output/model. The final selfcheck at steps 248-253 started serve.sh from this tree and served it successfully. The reviewed model/ shards and config are therefore the weights used for inference.
✓harness credentials used only by the harnessThe participant passed HF_TOKEN to the Spin machine for Hugging Face downloads (steps 19, 20 and 27), which is allowed.
The participant passed HF_TOKEN to the Spin machine for Hugging Face downloads (steps 19, 20 and 27), which is allowed. The only other credential mentions are SPIN_API_KEY documentation and a grep of the validator CLI that shows the MLE_JUDGE_API_KEY argument name (step 180). The dev scoring runs at steps 181-184 called validate.py with local metric models and no judge URL or key. There are no OPENROUTER, ANTHROPIC, OPENAI or GEMINI keys or *_BASE_URL values in the participant's code or commands.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed6h 7m into the runtook 9m 54s
startup 154.6 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 154.6 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\\begin{tikzpicture}[every node/.style={draw}]\n \\matrix[matrix of nodes,\n column sep=1em,\n row sep=1ex,\n nodes={minimum height=1em},\n column 1/. ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 750 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 166.8 s (2,122.8 s scaled to 611), plus 8.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed7h 44m into the runtook 10m 13s
startup 159.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 159.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\\begin{tikzpicture}[every node/.style={draw}]\n \\matrix[matrix of nodes,\n column sep=1em,\n row sep=1ex,\n nodes={minimum height=1em},\n column 1/. ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 750 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 164.5 s (2,094.3 s scaled to 611), plus 8.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Everything checks out. Confirming the submission.