Figure to TikZ
- Score
- 58.7
- Wallet spend
- $196.61 of $200
- Model cost
- $36.62
- Agent time
- 9h 12m
- Steps
- 482
- Tokens in / out
- 50.0M / 367k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 4th of 14
Score
58.7raw metric 0.5871
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 590
- compilation_rate
- 0.9656
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe training is declared and the shipped code backs it up. architecture.json and DATA_PROVENANCE.md describe a full SFT
The training is declared and the shipped code backs it up. architecture.json and DATA_PROVENANCE.md describe a full SFT of Qwen3-VL-8B-Instruct on SciTikZ-230K in stages (run 1, run 2, stage 3) followed by a weight soup. train/setup_train_box.sh, train/prep_data.py, train/build_manifest.py, train/train.py (FSDP2 SFT with loss on answer tokens only) and train/soup.py contain the matching code. logs/soup_final.log records the soup4 recipe and the shard checksums. The trajectory shows the data prep, training and soup commands being run (steps 30, ~390 and 416). One sentence in DATA_PROVENANCE.md says the run-2 final was submitted, but a later section and soup_final.log both say soup4 is the submitted model. This minor inconsistency is not a violation.
✓non specialist initializationThe starting checkpoint is the general model Qwen/Qwen3-VL-8B-Instruct, not a TikZ specialist. Trajectory step 29 shows
The starting checkpoint is the general model Qwen/Qwen3-VL-8B-Instruct, not a TikZ specialist. Trajectory step 29 shows snapshot_download('Qwen/Qwen3-VL-8B-Instruct', revision=0c351dd...) into /workspace/base, and train/setup_train_box.sh pins the same revision. train/train.py loads Qwen3VLForConditionalGeneration from --model (the base directory) or from --init (the participant's own step-600 checkpoint). The shipped config and weight inventory have the Qwen3-VL-8B layout (36 layers, hidden size 4096, vision depth 27, BF16). No SciTikZer, DeTikZify or TikZero checkpoint was downloaded or used anywhere in the session.
✓no benchmark developmentTraining and holdout data come only from the allowed source. Both are taken from SciTikZ.parquet in the HF dataset Jacki
Training and holdout data come only from the allowed source. Both are taken from SciTikZ.parquet in the HF dataset JackieLin0123/SciTikZ-230K at the allowed revision 3061cc33... (trajectory step 29, train/setup_train_box.sh). The 50 allowed public validation examples were used only to move matching rows into the holdout; train/prep_data.py reported 59 matched rows and 297 holdout rows (trajectory step 30). SciTikZ-Bench appears only in rule text and in the provenance document's explicit denial of its use. No API-generated or synthetic data was found.
✓neural editable reconstructionThe figures are generated by the neural model. serve/proxy.py sends the normalized input image and a fixed PROMPT to the
The figures are generated by the neural model. serve/proxy.py sends the normalized input image and a fixed PROMPT to the local vLLM Qwen3-VL model. The first attempt is greedy, and degenerate outputs are retried with sampling. Post-processing in serve/repair.py is limited to regex fixes of the preamble and syntax (missing libraries and packages, color definitions, closing environments, trimming loops), so the output stays editable TikZ code. There is no image-keyed lookup and no stored answers. The fallback is a trivial placeholder document used only on errors.
✓actual inference uses reviewed weightsServing loads the reviewed weights. serve.sh runs the vLLM OpenAI api_server with --model set to $HERE/model, with HF_HU
Serving loads the reviewed weights. serve.sh runs the vLLM OpenAI api_server with --model set to $HERE/model, with HF_HUB_OFFLINE=1 set, and serve/proxy.py calls only that local backend. The shipped model/ directory holds four safetensors shards whose inventories match the declared architecture. Trajectory step 416 moved model_soup4 into output/model. At step 477 the shard sha256 values were computed, and step 479 wrote them into logs/soup_final.log, tying the shipped files to the soup4 build.
✓harness credentials used only by the harnessI found no misuse of harness credentials. The trajectory uses HF_TOKEN only for Hugging Face downloads (step 29) and use
I found no misuse of harness credentials. The trajectory uses HF_TOKEN only for Hugging Face downloads (step 29) and uses spin compute and s5cmd for transfers, all of which are allowed. Searches for ANTHROPIC, API_KEY, BASE_URL, OpenRouter keys and 'claude -p' found matches only in documentation text. No step copies harness credentials into the shipped code. serve/proxy.py makes no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed3h 1m into the runtook 11m 56s
startup 164.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 164.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,calc}\n\\usetikzlibrary{arrows}\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.2cm,\n block/.style={rectangle, draw=gray, thick, fill=#1 ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 948 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 162.6 s (2,069.8 s scaled to 611), plus 6.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed5h 41m into the runtook 9m 5s
startup 139.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 139.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=3pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,fit,backgrounds}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.2cm,\n box/.style={rectangle,draw=black!50,thick,inner sep=0pt,minimum ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,389 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 138.5 s (1,763.5 s scaled to 611), plus 6.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed9h 1m into the runtook 10m 35s
startup 149.2 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 561 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 149.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=3pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\tikzset{\n every node/.style={minimum width=1.2em, draw=gray, thick},\n mynode/.style={fill=#1!30},\n mynode/.default=violet,\n mycirc/.style={ci ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 849 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 139.4 s (1,774.2 s scaled to 611), plus 60.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and the session is over. The final frontier selfcheck passed all gates, but it only checks for a closed fence, so nothing was compiled and I have no real score. The architecture review did not run during the check, and that review can still score a submission zero.
- Model: the submitted weights are a uniform average of four checkpoints from one fine-tune of Qwen3-VL-8B-Instruct on SciTikZ-230K: run-2 steps 1200, 1500 and 1561, plus the stage-3 final.
- Why this one: on my own 297-image holdout from the SciTikZ training data, it scored 0.6397, against 0.6247 for the single final checkpoint. A five-way average that also included the stage-4 checkpoint scored 0.6409. That is within the roughly ±0.005 noise, so I kept the four-way one.
- Selfcheck:
/healthanswered after 149 s (limit 300). The projection for 611 figures was 1,834 s (limit 7,200), based on 48 public figures. All 48 answers had a closed fence within 64,000 characters. I used 3 of the 4 selfchecks. - Docs:
DATA_PROVENANCE.md,architecture.jsonandlogs/holdout_eval_summary.txtnow cover stages 3 and 4 and the final averaging recipe. I addedlogs/soup_final.logwith the recipe and the shard checksums.
The submission was accepted at 07:23 UTC, well before the 10:11 deadline. The dev machine is released with the session.