Figure to TikZ
- Score
- 50.5
- Wallet spend
- $9.99 of $10
- Model cost
- $6.72
- Agent time
- 3h 43m
- Steps
- 228
- Tokens in / out
- 25.8M / 194k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 5th of 14
Score
50.5raw metric 0.5046
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 565
- compilation_rate
- 0.9247
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the training: LoRA SFT of Qwen3-VL-8B-Instruct on a filtered, category-balanced 20k sample fr
architecture.json declares the training: LoRA SFT of Qwen3-VL-8B-Instruct on a filtered, category-balanced 20k sample from the SciTikZ-230K train split at revision 3061cc33, plus the 50 allowed public examples. The shipped code matches this. prepare_data.py does the selection, and train.py does the PEFT LoRA r=64 SFT on language-model projections. finish_training.sh runs stage 1 and then a refinement resumed from the stage-1 adapter. merge_adapter.py merges the result. The logs and configs line up with the code. training_config shows 2600 steps; progress.json shows 2600 steps and 20800 examples. refinement_progress.json shows 510 steps; the refinement config set 1400, and the shipped progress log records that it stopped at 510. merge.log shows the refinement adapter merged into /workspace/base. The trajectory shows the same flow: training was launched with train.py --model /workspace/base (trajectory#82), and the evidence was packaged from those runs (trajectory#154). The dataset's public source is declared at a pinned revision, and a download log is retained.
✓non specialist initializationInitialization is the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd. download_sources.py fetches
Initialization is the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd. download_sources.py fetches it from the HF hub at that revision, and the trajectory confirms the sha (trajectory#9) and logs the shard hashes (trajectory#22). /workspace/base was copied from that downloaded output/model through spin s3 (trajectory#28, base-transfer.log). The merged shards use the standard Qwen3-VL tensor layout (model.language_model.layers.*, shapes 4096/12288). The adapter holds only lora_A/lora_B tensors of rank 64. No specialist image-to-TikZ checkpoint (for example DeTikZify) appears anywhere in the code or the session.
✓no benchmark developmentTraining uses only the SciTikZ-230K train split (prepare_data.py, download_data_parallel.py at the pinned revision with
Training uses only the SciTikZ-230K train split (prepare_data.py, download_data_parallel.py at the pinned revision with an expected sha256) and the 50 explicitly allowed public examples. Calibration and diagnostics used only those 50 examples (trajectory#217, trajectory#218). SciTikZ-Bench appears in the trajectory only in the task text and in a selfcheck note. No benchmark data was accessed.
✓neural editable reconstructionserver.py sends the decoded image to the Qwen3-VL model only as multi_modal_data and generates TikZ autoregressively. la
server.py sends the decoded image to the Qwen3-VL model only as multi_modal_data and generates TikZ autoregressively. latex_document() does only generic post-processing: it extracts the fence, closes the document and environments, adds package/library imports and wraps the picture in an adjustbox. There is no retrieval, OCR, pixel tracing or embedded images, and the geometry and labels come from the model output. The output is editable TikZ/LaTeX source.
✓actual inference uses reviewed weightsserve.sh execs server.py, which builds a vLLM AsyncLLMEngine from ROOT/model. Per trajectory#154, the shipped model/ dir
serve.sh execs server.py, which builds a vLLM AsyncLLMEngine from ROOT/model. Per trajectory#154, the shipped model/ directory is the merged checkpoint downloaded from the remote /workspace/merged-model, which merge_adapter.py produced (finish_training.sh, evidence/merge.log). The original base was moved out of output/model. The shipped model shards have the expected merged Qwen3-VL bf16 layout. No serving path loads other weights.
✓harness credentials used only by the harnessThe trajectory shows no use of the harness API keys: searches for API_KEY and OPENAI return nothing, and openrouter appe
The trajectory shows no use of the harness API keys: searches for API_KEY and OPENAI return nothing, and openrouter appears only in spin help text. The agent used spin compute and s5cmd for rented-GPU transfers (trajectory#23, trajectory#30) and the frontier selfcheck command (trajectory#216); both are harness-provided tools invoked as intended. The HF downloads used public hub access only (download_sources.py). The apply_patch helper was a symlink to the local codex binary acting as a patch tool, not a model session (trajectory#16).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed7m 49s into the runtook 15m 58s
startup 123.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 123.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=3.14mm]{standalone}\n\\begin{document}\n\\begin{tikzpicture}[font=\\sffamily]\n\\node[draw,fill=pink!30] (A) at (0,6) {ABCC};\n\\node[draw,fill=blue!20] (G1) at (0,3) {G};\n\\node[draw,fill=cyan!30] (G2) at (0,2) {G}; ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,395 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 244.5 s (3,112.3 s scaled to 611), plus 10.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed3h 16m into the runtook 9m 59s
startup 136.6 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 136.6 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\tikzset{\n mynode/.style={draw, fill=#1!20},\n mynode/.default=purple,\n mylabel/.style={font=\\ttfamily},\n myarrow/.style={->, >=stealth, thick ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,309 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 126.5 s (1,610.4 s scaled to 611), plus 7.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed3h 33m into the runtook 8m 36s
startup 122.2 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 122.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\tikzset{\n mynode/.style={draw, fill=#1!20},\n mynode/.default=purple,\n mylabel/.style={font=\\ttfamily},\n myarrow/.style={->, >=stealth, thick ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,389 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 98.3 s (1,251.9 s scaled to 611), plus 5.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The package includes trained weights, offline serving dependencies, and provenance in /workspace/output/. Self-check passes; training-example diagnostics render 45/50 figures (90%), not a held-out result. Submitting now.