Figure to TikZ
- Score
- 39.5
- Wallet spend
- $9.92 of $10
- Model cost
- $5.63
- Agent time
- 3h 41m
- Steps
- 271
- Tokens in / out
- 27.0M / 199k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 12th of 14
Score
39.5raw metric 0.3952
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 454
- compilation_rate
- 0.7430
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe training method is declared and matches the shipped code. architecture.json names the base model as Qwen3-VL-8B-Inst
The training method is declared and matches the shipped code. architecture.json names the base model as Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. train_sft.py runs full-LLM SFT with the vision tower frozen, using AdamW8bit and a cosine LR schedule. The training data is 11,000 compile-checked SciTikZ-230K examples, and data_manifest.json, DATA.md and prep_data.py declare where they came from. The trajectory shows the dataset was downloaded from the public HF repo JackieLin0123/SciTikZ-230K at a pinned revision, and the training run was launched at trajectory#216. Two inaccuracies, neither a violation: (1) DATA.md says the full planned schedule ran, but train.log and train_meta.json show the run stopped at about 208 of 216 steps on the time budget. (2) The base model was downloaded without a revision flag, and the revision SHA in architecture.json was read back from that download afterwards.
✓non specialist initializationtrain_sft.py starts from Qwen3VLForConditionalGeneration.from_pretrained on the general-purpose Qwen/Qwen3-VL-8B-Instruc
train_sft.py starts from Qwen3VLForConditionalGeneration.from_pretrained on the general-purpose Qwen/Qwen3-VL-8B-Instruct checkpoint. The trajectory shows it was downloaded straight from that HF repo. The shipped weights have the standard Qwen3-VL-8B bf16 tensor layout, and config.json declares qwen3_vl. There is no sign the model started from a TikZ- or scientific-figure-specialist checkpoint.
✓no benchmark developmentAll training data comes from SciTikZ-230K. prep_data.py removed the 50 public validation examples by SHA-1 match (32 hit
All training data comes from SciTikZ-230K. prep_data.py removed the 50 public validation examples by SHA-1 match (32 hits), and train_meta.json records the dev exclusion. Using the public validation data is allowed anyway. Neither the trajectory nor the code shows any download or use of SciTikZ-Bench.
✓neural editable reconstructionserve.sh serves the fine-tuned Qwen3-VL vision-language model through vLLM's OpenAI-compatible server, with one image pe
serve.sh serves the fine-tuned Qwen3-VL vision-language model through vLLM's OpenAI-compatible server, with one image per prompt. Training taught the model to emit editable TikZ/LaTeX code inside a ```latex block. There is no retrieval, lookup table or non-neural shortcut in the serving path.
✓actual inference uses reviewed weightsserve.sh runs vLLM offline (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1) with --model "$ROOT/model". That directory holds t
serve.sh runs vLLM offline (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1) with --model "$ROOT/model". That directory holds the four reviewed safetensors shards, the index file and config.json. No other inference code and no external API calls exist.
✓harness credentials used only by the harnessThe trajectory shows no use of harness or judge model-API credentials such as ANTHROPIC, OPENAI or MLE_JUDGE_API_KEY. Th
The trajectory shows no use of harness or judge model-API credentials such as ANTHROPIC, OPENAI or MLE_JUDGE_API_KEY. The judge-key argument appears only in grep output of cli.py, and validation was run without a judge. Other credentials are allowed under policy: the HF_TOKEN used for downloads, and the Spin transfer credentials (session.json from `spin transfer credentials`) that export.sh used for R2 uploads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed1h 5m into the runtook 27m 23s
startup 189.8 s of 300 s; 0/1 gates passed; failed 611 figures in 7200 s: projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 564.7 s share of the deadline after a 12.4 s first figure (Batch exceeded 564.658s deadline; score is ze
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 189.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=3.14mm]{standalone}\n\\usetikzlibrary{positioning,calc}\n\\begin{document}\n\\begin{tikzpicture}[node distance=0.5cm,font=\\sffamily]\n\\node[draw,fill=red!20] (A) {ABCC};\n\\node[draw,fill=blue!20,below=of A] (B) {XY ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,231 characters of LaTeX
- ✗611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 564.7 s share of the deadline after a 12.4 s first figure (Batch exceeded 564.658s deadline; score is zero)
- 2passed3h 13m into the runtook 13m 37s
startup 162.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 162.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,fit,calc}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.5cm,\n every node/.style={draw,fill=gray!50,inner sep=0pt,minimum width=1cm,m ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 915 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 345.4 s (4,396.5 s scaled to 611), plus 10.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 36 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Dev score: compile 27/50 (54%), compiled-only SigLIP 0.935 / CLIP 0.903 / SSIM 0.67 / DreamSim-normalized 0.81 — an estimated scalar ≈ 0.29–0.33 (the real eval's 16384-token budget gives the 11 truncated answers 2.7× more room than my dev cap, so the true number should land higher). GPU budget is now exhausted, so I'm finalizing the submission as-is: