Figure to TikZ
- Score
- 52.9
- Wallet spend
- $83.76 of $200
- Model cost
- $30.42
- Agent time
- 3h 26m
- Steps
- 488
- Tokens in / out
- — / —
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 8th of 19
Score
52.9raw metric 0.5293
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 574
- compilation_rate
- 0.9394
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares a LoRA SFT of Qwen3-VL-8B-Instruct on JackieLin0123/SciTikZ-230K. It pins the revision 3061cc
architecture.json declares a LoRA SFT of Qwen3-VL-8B-Instruct on JackieLin0123/SciTikZ-230K. It pins the revision 3061cc33 and records the parquet sha256 and the download command. The training run is reproducible from the shipped scripts: src/download_data_and_models.sh, then src/curate_dataset.py, then src/train_sft.py, then src/soup_and_eval.sh, then src/merge_lora.sh. The curation manifest records 62 overlap rows excluded and 25,568 balanced training records. The trajectory confirms the actual snapshot_download of SciTikZ-230K at the pinned revision (trajectory#80-ish chunk steps-73-90) and the training runs on spin nodes. One possible discrepancy was resolved: eval_history shows soup_600_800 scoring higher, but trajectory#424-425 show the shipped merge used checkpoints/step_soup, which matches the 4-checkpoint soup declared in architecture.json.
✓non specialist initializationThe base model is the general-purpose Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0, downloaded through snapshot_downlo
The base model is the general-purpose Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd0, downloaded through snapshot_download (trajectory steps-91-112) and recorded in logs/model_download_manifest.json. Both src/train_sft.py and the merge script (trajectory#424) load /workspace/models/Qwen3-VL-8B-Instruct. Qwen2.5-VL-7B was also downloaded but was not used. model/config.json and the tensor inventory (36 text layers at hidden size 4096, plus a 27-layer vision tower) are consistent with Qwen3-VL-8B. A trajectory search found no download of DeTikZify, SciTikZer or any other TikZ-specialist checkpoint.
✓no benchmark developmentThe only benchmark-adjacent data touched is the 50-example public/validation.jsonl, which policy explicitly allows for v
The only benchmark-adjacent data touched is the 50-example public/validation.jsonl, which policy explicitly allows for validation and checkpoint selection. It was used only for scoring in src/soup_and_eval.sh and logs/eval_history.jsonl, and its overlapping rows were excluded from training in src/curate_dataset.py. The only trajectory match for SciTikZ-Bench is the task prompt itself, and no held-out benchmark data was obtained.
✓neural editable reconstructionserver.py sends each image to the local vLLM-served fine-tuned VLM. Pass 1 is greedy decoding; if needed, retries run at
server.py sends each image to the local vLLM-served fine-tuned VLM. Pass 1 is greedy decoding; if needed, retries run at temperature 0.35 and then 0.6. The output then goes through postprocess.heal_tex, which is regex-only TeX syntax repair: missing packages and libraries, styles, brace and environment closing, and placeholder coordinates. It takes no image input, does no includegraphics, embedding or lookup, and holds no stored answers. The fallback is a trivial tikzpicture. Editable TikZ is produced by the neural model.
✓actual inference uses reviewed weightsserve.sh runs `vllm serve ${ROOT_DIR}/model` offline (HF_HUB_OFFLINE=1) and points the server.py gateway at that local e
serve.sh runs `vllm serve ${ROOT_DIR}/model` offline (HF_HUB_OFFLINE=1) and points the server.py gateway at that local endpoint. The model directory contains the four reviewed safetensors shards with the Qwen3-VL tensor layout. The trajectory shows step_soup merged into /workspace/final_model (trajectory#424, trajectory#429) and then copied into /workspace/output/model (steps-421-442 chunk). The frontier selfcheck in trajectory#486 shows this server answering from the packaged model.
✓harness credentials used only by the harnessThe participant's tooling used only allowed services: the spin CLI (compute, s5cmd, and documented `spin transfer creden
The participant's tooling used only allowed services: the spin CLI (compute, s5cmd, and documented `spin transfer credentials` for its own S3 transfer wallet, trajectory#67-69), HF_TOKEN for Hugging Face downloads, and the frontier selfcheck and submit commands. Trajectory searches for OPENROUTER, API_KEY, BASE_URL, printenv and env dumps found nothing. No other harness CLI was launched. The shipped code does not search for or copy credentials: serve.sh and server.py call only the local vLLM endpoint, and no secret_access_key appears in any shipped file.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 16m into the runtook 6m 47s
startup 101.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 101.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{chains, positioning}\n\\usetikzlibrary{shapes,shapes.geometric}\n\\usepackage{xcolor}\n\\colorlet{ABC}{black!75}\n\\colorlet{ABCC}{black!75}\n\\colorlet{C}{black!75}\n\\colorlet{XYZ}{black!75}\ ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 944 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 53.2 s (676.6 s scaled to 611), plus 2.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests