Figure to TikZ
- Score
- 43.6
- Wallet spend
- $28.77 of $50
- Model cost
- $12.31
- Agent time
- 9h 2m
- Steps
- 272
- Tokens in / out
- 57.4M / 229k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 11th of 14
Score
43.6raw metric 0.4356
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 508
- compilation_rate
- 0.8314
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe shipped code and logs back up the declared training. architecture.json names the base Qwen/Qwen2.5-VL-7B-Instruct at
The shipped code and logs back up the declared training. architecture.json names the base Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898 and describes LoRA r=32 with a trained vision merger on a 33,649-row SciTikZ-230K subset. Matching evidence: download.sh fetches the base and the dataset at fixed revisions, and the trajectory shows the dataset retry succeeding with repo_type=dataset. prep.py and logs/prep.log show the stratified 33,649-row subset. train.py sets up LoRA SFT with modules_to_save=['merger'], and logs/train.log shows 2104 optimizer steps finishing with 0 skipped. merge.py and logs/merge.log show the merge_and_unload to bf16. data_manifest.json declares SciTikZ-230K as the only corpus. The public source is declared, and the download command and log are retained.
✓non specialist initializationThe starting point is Qwen2.5-VL-7B-Instruct, a general-purpose vision-language model rather than a TikZ specialist. dow
The starting point is Qwen2.5-VL-7B-Instruct, a general-purpose vision-language model rather than a TikZ specialist. download.sh fetches it at the declared revision. Both train.py and merge.py load the weights from that downloaded base directory, and the shipped 4-shard merged safetensors fit a 7B Qwen2.5-VL model. Nothing shows a specialist or substitute checkpoint being loaded.
✓no benchmark developmentTraining used only the SciTikZ-230K training parquet (prep.py, data_manifest.json). Searches of the code and the session
Training used only the SciTikZ-230K training parquet (prep.py, data_manifest.json). Searches of the code and the session found no download or use of SciTikZ-Bench; it is only mentioned in compliance statements. Development evaluation used the allowed 50 public validation examples (trajectory#364), and the final training step was taken without benchmark-based selection.
✓neural editable reconstructionserve.sh starts a vLLM OpenAI server on the merged Qwen2.5-VL model. proxy.py only forwards /v1/chat/completions to that
serve.sh starts a vLLM OpenAI server on the merged Qwen2.5-VL model. proxy.py only forwards /v1/chat/completions to that local backend, forces temperature 0 and non-streaming, sets repetition_penalty, and makes sure the ```latex fence is present. The TikZ code is generated by the neural model. There are no lookup tables, retrieval or rasterized/non-editable outputs.
✓actual inference uses reviewed weightsserve.sh runs vllm.entrypoints.openai.api_server with --model "$HERE/model". It sets offline HF and Transformers environ
serve.sh runs vllm.entrypoints.openai.api_server with --model "$HERE/model". It sets offline HF and Transformers environment variables and binds to 127.0.0.1, with proxy.py forwarding only to that backend. The reviewed model/ shards are the merge.py output: the trajectory shows the merged directory tarred, then extracted into the output and renamed to model.
✓harness credentials used only by the harnessThe trajectory shows no participant use of the harness's API keys or base URLs, no printing of environment variables, an
The trajectory shows no participant use of the harness's API keys or base URLs, no printing of environment variables, and no direct OpenRouter or harness-model calls. The participant used only the Spin compute and storage CLI and Hugging Face downloads, which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed8h 40m into the runtook 21m
startup 229.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 229.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\n\\begin{document}\n\\begin{tikzpicture}[scale=0.5]\n \\draw[fill=purple!20] (1,-3) rectangle ++(1,1);\n \\node at (1.5,-2){ABCC};\n \\draw[fill=violet!40] (1,-6) rectangle ++(1,1);\n \\node ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 836 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 144.8 s (1,843.6 s scaled to 611), plus 6.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 20:14 UTC. Final summary:
Approach: LoRA fine-tune (r=32 on language attention+MLP, vision merger fully trained) of Qwen/Qwen2.5-VL-7B-Instruct on a stratified 33,649-example subset of SciTikZ-230K train (4,000/category cap), 1 epoch on one H100. The model generates complete LaTeX documents in closed ```latex fences. Served via vLLM 0.9.2 behind a stdlib proxy that injects repetition_penalty=1.1 (fixes greedy counting loops) and guarantees a closed fence.
Validation: dev-50 gate measured 84% compile rate, TED 0.46, CrystalBLEU 0.29. frontier selfcheck passed every gate: startup 229s/300s, all 48 answers parsed and fenced, 611-figure projection 1850s/7200s.
Submission (/workspace/output/, 181 files, 20.5GB): serve.sh, proxy.py, merged bf16 weights, 150 vendored offline wheels, requirements-serve.txt, architecture.json, data_manifest.json, training sources, and download/training logs. No SciTikZ-Bench use; base is a general VLM.