Figure to TikZ
- Score
- 57.0
- Wallet spend
- $40.52 of $50
- Model cost
- $22.34
- Agent time
- 9h 12m
- Steps
- 327
- Tokens in / out
- 79.9M / 228k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 3rd of 14
Score
57.0raw metric 0.5701
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 590
- compilation_rate
- 0.9656
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares a two-phase LoRA SFT (r=128, alpha=256) of Qwen3-VL-8B-Instruct on SciTikZ-230K. Phase 1 used
architecture.json declares a two-phase LoRA SFT (r=128, alpha=256) of Qwen3-VL-8B-Instruct on SciTikZ-230K. Phase 1 used 60k rows at lr 1e-4 for 2167 steps. Phase 2 continued from the phase-1 adapter on rows 60k-90k, minus the held-out rows, plus a replay, at lr 4e-5 for 1125 steps. The code matches this: train/prep_data.py builds the pool, train/train_sft.py supports --init_adapter and --exclude_ids, and train/merge_lora.py merges the adapter. The logs logs/train_main.log, logs/train_cont.log and logs/train_log.jsonl agree with the code, and the session shows the merged continuation model placed in output/model (trajectory#318).
✓non specialist initializationTraining starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd0. provenance/download_data
Training starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd0. provenance/download_data.sh downloads it with snapshot_download, and the session shows the download in trajectory#23 and #25. The shipped checkpoint shards contain the standard Qwen3-VL tensor layout, which fits a merged LoRA on that base. Nothing shows initialization from a TikZ-specialist checkpoint.
✓no benchmark developmentThe training data is SciTikZ-230K at pinned revision 3061cc33, a declared public dataset downloaded in provenance/downlo
The training data is SciTikZ-230K at pinned revision 3061cc33, a declared public dataset downloaded in provenance/download_data.sh (trajectory#12, #23). The 50 public validation IDs supplied with the task are excluded from training in train/prep_data.py. The held-out rows from make_heldout come out of the same corpus (trajectory#93). The validation set was used only for local evaluation and decoding selection. Nothing shows the benchmark test data being fetched or trained on.
✓neural editable reconstructionsrc/server.py runs the fine-tuned vision-language model through vLLM to generate TikZ code: one greedy candidate plus se
src/server.py runs the fine-tuned vision-language model through vLLM to generate TikZ code: one greedy candidate plus several sampled candidates. src/candidates.py picks among them with lint checks and n-gram MBR consensus. src/postprocess.py only extracts code fences and closes environments, and src/repair.py only adds missing packages, TikZ libraries and color definitions. Nothing in src embeds the input image with includegraphics or returns a raster copy, so the output is editable TikZ produced by the neural model.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$HERE/model, turns on offline mode and execs src/server.py. That script builds the vLLM AsyncLLM
serve.sh sets MODEL_DIR=$HERE/model, turns on offline mode and execs src/server.py. That script builds the vLLM AsyncLLMEngine from MODEL_DIR. model/model.safetensors.index.json and model/config.json describe the reviewed Qwen3-VL checkpoint shards. The base model first copied in as a placeholder was replaced by the merged continuation model (trajectory#318).
✓harness credentials used only by the harnessThe participant ran grep on the validator, and the output showed the name MLE_JUDGE_API_KEY (trajectory#92), but the ses
The participant ran grep on the validator, and the output showed the name MLE_JUDGE_API_KEY (trajectory#92), but the session never used that key. HF_TOKEN was used only to download the public dataset and base model (trajectory#25). The R2/Spin credentials were used only to move the participant's own artifacts (trajectory#312). Nothing shows harness-only credentials being used by the participant.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed36m 13s into the runtook 12m 7sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed49m 21s into the runtook 9m 22s
startup 145.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 145.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=3.14mm]{standalone}\n\\usetikzlibrary{positioning,calc}\n\\begin{document}\n\\begin{tikzpicture}[node distance=0.5cm,font=\\sffamily]\n\\node[draw,fill=red!20] (A) {ABCC};\n\\node[draw,fill=blue!20,below=of A] (B) {XY ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,231 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 134.2 s (1,708.5 s scaled to 611), plus 7.7 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed5h 56m into the runtook 9m 38s
startup 145.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 145.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{chains,shapes.multipart}\n\n\\begin{document}\n\\begin{tikzpicture}[\n start chain=going below,\n node distance=0pt,\n every node/.style={draw=gray, fill=whit ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 795 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 103.6 s (1,319.3 s scaled to 611), plus 17.6 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed8h 58m into the runtook 9m 44s
startup 135.2 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 563 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 135.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=2pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,calc}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0pt,\n every node/.style={draw=black!60, fill=white, minimum width=1.5em, minimum h ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,658 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 89.6 s (1,141.1 s scaled to 611), plus 34.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The local package now holds exactly the evaluated continuation model (checksums match). I'm updating the docs, then running the final selfcheck.