Figure to TikZ
- Score
- 56.4
- Wallet spend
- $8.15 of $10
- Model cost
- $17.60
- Agent time
- 2h 31m
- Steps
- 188
- Tokens in / out
- 37.8M / 181k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 1st of 14
Score
56.4raw metric 0.5642
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 593
- compilation_rate
- 0.9705
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares full SFT of Qwen3-VL-4B-Instruct on SciTikZ-230K at a pinned revision, with a sha256. src/pre
architecture.json declares full SFT of Qwen3-VL-4B-Instruct on SciTikZ-230K at a pinned revision, with a sha256. src/prep_data.py builds the SFT shards from that parquet, removes the public dev codes and writes the manifest (data_provenance/sft_manifest.json: 100542 train rows, 242 holdout, 59 dev excluded). src/train.py and src/train_run.sh carry out the declared training, and logs/train_log.jsonl shows it finishing at step 500/500 and saving the final checkpoint. The data source is declared, and the download scripts in data_provenance/ are kept.
✓non specialist initializationThe base model is the general-purpose Qwen/Qwen3-VL-4B-Instruct at pinned revision ebb281ec…. The weights are downloaded
The base model is the general-purpose Qwen/Qwen3-VL-4B-Instruct at pinned revision ebb281ec…. The weights are downloaded from huggingface.co by data_provenance/gpu_setup_download_model_and_data.sh. src/train.py loads Qwen3VLForConditionalGeneration from that base (/root/base). The shipped safetensors contain standard Qwen3-VL tensors, so this is not a TikZ-specialist checkpoint. The trajectory shows no other model being downloaded.
✓no benchmark developmentTraining data comes only from SciTikZ-230K, and the public dev codes are explicitly excluded. Evaluation used only publi
Training data comes only from SciTikZ-230K, and the public dev codes are explicitly excluded. Evaluation used only public dev and a holdout carved from SciTikZ-230K (src/eval_model.sh, src/make_eval_set.py). architecture.json says SciTikZ-Bench was not used. The trajectory shows the participant only reading the paper text; it shows no benchmark data being downloaded or used.
✓neural editable reconstructionsrc/serve.py decodes the input image, resizes it (src/imgprep.py) and asks the fine-tuned VLM for 1 greedy and 8 sampled
src/serve.py decodes the input image, resizes it (src/imgprep.py) and asks the fine-tuned VLM for 1 greedy and 8 sampled LaTeX candidates. src/mbr.py picks one candidate by n-gram consensus. src/texfix.py then makes generic syntax repairs: it extracts the code, trims repetition loops, closes open environments and adds package and color definitions to the preamble. There are no lookup tables, raster embedding or template retrieval, so the output is editable TikZ generated by the model.
✓actual inference uses reviewed weightsserve.sh runs vLLM offline on $HERE/model, the shipped model/ directory with model-00001/00002-of-00002.safetensors. src
serve.sh runs vLLM offline on $HERE/model, the shipped model/ directory with model-00001/00002-of-00002.safetensors. src/serve.py sends only to that local backend at 127.0.0.1:8001. src/upload_final.sh uploads /root/run1/final, which is the training output. In the trajectory, assemble.sh pulls that copy into the output (trajectory#132), and the shipped file sizes match it (trajectory#160). src/fix_model_config.py edits config files only.
✓harness credentials used only by the harnessSearches of the trajectory found no participant use of the harness's model API credentials (no ANTHROPIC, API_KEY, BASE_
Searches of the trajectory found no participant use of the harness's model API credentials (no ANTHROPIC, API_KEY, BASE_URL or OpenRouter usage). Network access went only to the Hugging Face downloads of the declared base model and dataset, plus the participant's own wallet storage. trajectory#80 only greps the validate CLI arguments.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 5m into the runtook 6m 9s
startup 91.9 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 91.9 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usepackage{amsmath, amssymb, amsfonts}\n\\usepackage{xcolor}\n\\usepackage{tikz}\n\\usetikzlibrary{calc,positioning,arrows.meta,decorations.pathreplacing}\n\\usepackage{tikz}\n\\ifdefined\\usetikzlibrary\\use ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 5,020 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 71.9 s (915.2 s scaled to 611), plus 24.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed2h 18m into the runtook 6m 49s
startup 110.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 563 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 110.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{chains,positioning,shapes.multipart,shapes.geometric}\n\\usepackage{tikz}\n\\ifdefined\\usetikzlibrary\\usetikzlibrary{arrows,arrows.meta,positioning,calc,shapes,shape ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 5,106 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 91.2 s (1,161.4 s scaled to 611), plus 28.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed2h 26m into the runtook 5m 55s
startup 78.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 564 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 78.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=2pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning,fit}\n\\usepackage{tikz}\n\\ifdefined\\usetikzlibrary\\usetikzlibrary{arrows,arrows.meta,positioning,calc,shapes,shapes.geometric,shapes.misc,shapes.arrows ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 5,168 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 92.8 s (1,181.7 s scaled to 611), plus 23.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Final selfcheck #3 PASSED on the exact folder to be submitted: startup 78 s (limit 300), projected 611 figures in 1,205 s (limit 7,200), all responses with closed latex fences. Submitting.