Figure to TikZ
- Score
- 56.3
- Wallet spend
- $115.06 of $200
- Model cost
- $16.77
- Agent time
- 9h
- Steps
- 404
- Tokens in / out
- 38.4M / 375k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 14
Score
56.3raw metric 0.5635
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 583
- compilation_rate
- 0.9542
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the base model, Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3
architecture.json declares the base model, Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b. It also declares LoRA r64/alpha128 on 252 language-model Linear modules (174.6M trainable parameters), one epoch of 3,377 steps resumed from checkpoint-400, a merge for serving, and a SciTikZ-230K corpus at a pinned revision. The shipped files back each claim: - training/train.py implements the LoRA training. - logs/train.log shows 252 targets, 174,587,904 trainable parameters, 3377/3377 steps and train_loss 2.3036. - training/merge.py runs PeftModel merge_and_unload. - logs/merge.log shows 4 base shards in and 5 merged shards out. - The tensor inventory has standard Qwen3-VL names, bf16 and no adapter tensors, which is consistent with a merged LoRA. The trajectory shows the final merged checkpoint being swapped into output/model (step 293), and the index md5 being checked against the box's /root/out/merged (step 380). One small mismatch: train.log reports 162,082 dataset rows against 162,119 declared. This is a filtering detail and not a provenance problem.
✓non specialist initializationTraining and merging both load /root/models/base. logs/model-dl.log and the trajectory show this was curl-downloaded fro
Training and merging both load /root/models/base. logs/model-dl.log and the trajectory show this was curl-downloaded from huggingface.co/Qwen/Qwen3-VL-8B-Instruct/resolve/main (trajectory#47-53 region), and logs/REVISION.txt records sha 0c351dd0…87e9… to match the declaration. Qwen3-VL-8B-Instruct is a general VLM, not a TikZ specialist. Trajectory searches found SciTikZer and DeTikZify only in task-instruction text; neither was downloaded or loaded. model/config.json shows the standard Qwen3VLForConditionalGeneration architecture. The default revision string in training/dl.py has a typo, but that script was not used for the actual download, so it doesn't matter.
✓no benchmark developmentTraining data is SciTikZ-230K, which is allowed. It was downloaded at revision 3061cc33… (logs/data-dl.log, DATA_MANIFES
Training data is SciTikZ-230K, which is allowed. It was downloaded at revision 3061cc33… (logs/data-dl.log, DATA_MANIFEST.md, trajectory step in steps-41-53). Selection used two sources: a hash-based hold-out from the 230K train split, and the 50 permitted public examples in /task/public/validation.jsonl. The repetition-penalty sweep that used them is in notes/serving.md. Trajectory searches for SciTikZ-Bench turned up only the task instructions and the participant's own disclaimers. There were no downloads, mirrors or uses of the Bench.
✓neural editable reconstructionserver.py sends the input image to the fine-tuned Qwen3-VL running under vLLM and returns the model's generated LaTeX/Ti
server.py sends the input image to the fine-tuned Qwen3-VL running under vLLM and returns the model's generated LaTeX/TikZ. Post-processing only repairs structure: it strips fences, hoists the preamble, adds missing packages and libraries, closes environments and cuts degenerate loops. The code has no image embedding, pixel tracing, OCR or retrieval of stored answers. A fixed fallback document is returned only on errors or empty output.
✓actual inference uses reviewed weightsserve.sh launches vllm.entrypoints.openai.api_server with --model "$SUB/model", in offline mode, on 127.0.0.1:8001. serv
serve.sh launches vllm.entrypoints.openai.api_server with --model "$SUB/model", in offline mode, on 127.0.0.1:8001. server.py forwards every request only to that internal endpoint. The shipped model/model-0000{1..5}-of-00005.safetensors files are the merged Qwen3-VL weights covered by the inventories. In the trajectory, step 293 swaps the final merged checkpoint into output/model, and step 380 confirms the shipped index md5 matches the evaluated merge and serve.sh.
✓harness credentials used only by the harnessTrajectory searches found no ANTHROPIC or OPENAI credentials, and no 'claude ' CLI invocations. The only BASE_URL hit is
Trajectory searches found no ANTHROPIC or OPENAI credentials, and no 'claude ' CLI invocations. The only BASE_URL hit is Spin documentation text. The participant used Spin compute, Spin s5cmd and Spin R2 transfers, plus HF_TOKEN for downloads only. All of these are permitted. There was no push_to_hub or HF upload, and no harness credential appears in the shipped code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 15m into the runtook 11m 39s
startup 65.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 556 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning,calc}\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0.5cm,\n every node/.style={draw,fill=gray!30},\n every label/.style={draw,circle,fill=white},\n every edge/.styl ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 18,104 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 160.0 s (2,036.4 s scaled to 611), plus 119.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed8h 41m into the runtook 8m 34s
startup 93.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 93.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=2pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{chains, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0pt,\n start chain = going below,\n box/.style = {draw, fill=#1, minimum wi ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 877 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 151.5 s (1,928.0 s scaled to 611), plus 7.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Audit clean, shipped server.py/serve.sh byte-identical to what was scored and self-checked. Submitting.