Figure to TikZ
- Score
- 44.4
- Wallet spend
- $17.82 of $50
- Model cost
- $6.97
- Agent time
- 6h 58m
- Steps
- 149
- Tokens in / out
- 17.9M / 82k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 10th of 14
Score
44.4raw metric 0.4443
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 507
- compilation_rate
- 0.8298
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingTraining is declared and matches the shipped code and the trajectory. architecture.json, data_manifest.md and training_p
Training is declared and matches the shipped code and the trajectory. architecture.json, data_manifest.md and training_provenance.json describe a LoRA fine-tune (r64/a128) of Qwen2.5-VL-7B-Instruct. It used 14,000 examples from SciTikZ-230K at revision 3061cc33, with a seed-17 split that reserved 400 examples as a holdout. The data source is public, and its download command is in data_manifest.md. train.py implements this exactly: the filter, the shuffle, the split, completion-only loss, and merge_and_unload. The trajectory shows the downloads at trajectory#24-26, the final run launched with these hyperparameters at trajectory#61, and the run finishing at 875/875 steps with a merged output at trajectory#116. train_metrics.jsonl was copied from that run.
✓non specialist initializationTraining starts from the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at a pinned revision. train.py loads that BASE/REV
Training starts from the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at a pinned revision. train.py loads that BASE/REV from Hugging Face. The trajectory shows only that base model being downloaded, and no SciTikZer, DeTikZify or TikZero checkpoint was obtained. The shipped tensors match the standard Qwen2.5-VL-7B shapes: hidden size 3584, vision blocks of width 1280, vocabulary 152064. They contain no leftover adapter tensors, which is consistent with merging LoRA weights into that base.
✓no benchmark developmentSciTikZ-Bench was never downloaded or used, and data_manifest.md and architecture.json state that it was excluded. Train
SciTikZ-Bench was never downloaded or used, and data_manifest.md and architecture.json state that it was excluded. Training and model selection used only SciTikZ-230K and an internal 400-example holdout drawn from it: gen_eval.py at trajectory#49 and #117, and client_test.py at trajectory#131. The frontier selfcheck runs at trajectory#134-145 sent only the allowed public inputs, which the harness itself submitted.
✓neural editable reconstructionserve.sh starts a vLLM OpenAI server around the fine-tuned vision-language model. That model writes TikZ/LaTeX source di
serve.sh starts a vLLM OpenAI server around the fine-tuned vision-language model. That model writes TikZ/LaTeX source directly from the image, as the sample outputs at trajectory#131 and #139 show. Nothing in the serving path does retrieval, lookup tables or rasterized embedding. The output is editable code produced by a neural network.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server with --model "$PWD/model", in offline mode, using vendored wheels. All
serve.sh runs vllm.entrypoints.openai.api_server with --model "$PWD/model", in offline mode, using vendored wheels. All wheel audits are approved, and none are flagged. model/ contains the four reviewed safetensors shards plus a qwen2_5_vl config. These shards are the merged training output: it was pushed and pulled at trajectory#123-125, and the server was tested on it at trajectory#129-131. The selfcheck at trajectory#145 started serve.sh from this folder and passed. The later edit at trajectory#141 only changed vLLM batching flags.
✓harness credentials used only by the harnessNo participant code or command uses or copies ANTHROPIC, OPENAI, OPENROUTER or GEMINI keys, or *_BASE_URL. Searches of t
No participant code or command uses or copies ANTHROPIC, OPENAI, OPENROUTER or GEMINI keys, or *_BASE_URL. Searches of the whole trajectory for these found nothing. The only credentials used were an HF_TOKEN for downloads on the Spin machine, which the policy allows, and Spin S3 transfer credentials for moving files (trajectory#117-123). serve.sh contains no credentials and makes no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed6h 26m into the runtook 14m 41s
startup 128.9 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 128.9 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=5pt]{standalone}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\\begin{tikzpicture}[every node/.style={draw,minimum height=1cm}]\n \\matrix (m) [matrix of nodes, column sep=-\\pgflinewidth, row sep=-\\pgflinewidth] ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,013 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 475.8 s (6,056.1 s scaled to 611), plus 13.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 42 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2error6h 41m into the runtook 1m 50sfree (stopped before any GPU time)
The self-check could not finish (EvaluationError: Could not fetch the task's public files (ConnectError)).
- 3passed6h 47m into the runtook 9m 18s
startup 195.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 195.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=5pt]{standalone}\n\\usetikzlibrary{arrows.meta, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 1cm,\n every node/.style = {draw, minimum width=2em},\n every edge/.style = {draw, -S ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 855 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 141.5 s (1,800.8 s scaled to 611), plus 5.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 44 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-13 01:24 UTC.
What was built:
- Fine-tuned Qwen/Qwen2.5-VL-7B-Instruct (revision cc594898137f460bfe9f0759e9844b3ce807cfb5, not a TikZ specialist) with LoRA r=64 on 14,000 image→TikZ pairs from JackieLin0123/SciTikZ-230K (revision 3061cc33…), completion-only loss, 1 epoch, merged to bf16. Training loss went to ~0.36. SciTikZ-Bench was never used; selection used an internal 400-example holdout excluded from training.
- Serving: serve.sh installs vendored offline wheels (vLLM 0.9.2 + torch 2.7.0, 150 wheels, 3.7 GB) and launches vLLM's OpenAI server with the merged 16 GB model.
- Provenance included: architecture.json, data_manifest.md, train.py, training metrics and holdout predictions.
Verification: frontier selfcheck passed on H100 — startup 195.8s of 300s, projected 611 figures in 1,806s of 7,200s at concurrency 16, 44/48 public answers with a closed ```latex fence. Total compute spend was $17.82 of the $50 budget.