Figure to TikZ
- Score
- 44.4
- Wallet spend
- $7.80 of $10
- Model cost
- $11.91
- Agent time
- 5h 58m
- Steps
- 234
- Tokens in / out
- 21.5M / 283k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 8th of 14
Score
44.4raw metric 0.4436
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 505
- compilation_rate
- 0.8265
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe shipped files declare the training and the code matches it. architecture.json and PROVENANCE.md describe LoRA SFT (r
The shipped files declare the training and the code matches it. architecture.json and PROVENANCE.md describe LoRA SFT (r=64, alpha=128, LM projections only, vision tower frozen via exclude '.*visual.*') on Qwen2.5-VL-7B-Instruct. Training ran in two relay phases, and the phase-2 adapters were then merged into a bf16 checkpoint with a uniform LoRA soup. scripts/train.py, scripts/run_relay.sh and scripts/merge_soup.py implement exactly this. The phase-2 adapter configs, train logs and trainer states (1045 steps each) are shipped. The trajectory shows the phase-2 relay training (trajectory#177) and the packaging of its soup into model/ (trajectory#190, trajectory#210).
✓non specialist initializationThe model starts from the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898…, downloaded from Hugging Fac
The model starts from the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898…, downloaded from Hugging Face during the session (trajectory#14). The phase-2 adapter configs show the plain base path /root/model_base. model/config.json and the safetensors inventories match standard Qwen2.5-VL-7B (28 layers, hidden size 3584, 32-layer vision tower). The trajectory shows no download of a specialist TikZ model (SciTikZer, DeTikZify or TikZero).
✓no benchmark developmentThe training data and the 440-row holdout come only from the allowed SciTikZ-230K corpus at pinned revision 3061cc33…. T
The training data and the 440-row holdout come only from the allowed SciTikZ-230K corpus at pinned revision 3061cc33…. The download is in trajectory#14, and scripts/prepare_data.py reads the local SciTikZ parquet. SciTikZ-Bench appears only in the task prompt and in provenance statements; the trajectory has no download or use of it. Development evals were run on figures from the corpus holdout (trajectory#192-194, trajectory#217).
✓neural editable reconstructionserve.sh starts a vLLM OpenAI-compatible server on the fine-tuned VLM. serve_proxy.py only forwards chat completions to
serve.sh starts a vLLM OpenAI-compatible server on the fine-tuned VLM. serve_proxy.py only forwards chat completions to localhost:8001. On top of that it only repairs the latex fence and applies the character cap, the max_tokens clamp, a timeout guard and empty fallbacks. There is no retrieval, OCR, pixel tracing or stored-answer lookup, so the model itself generates editable TikZ code.
✓actual inference uses reviewed weightsserve.sh passes --model "$HERE/model" to vLLM with HF_HUB_OFFLINE=1, and the proxy calls only that localhost backend. Th
serve.sh passes --model "$HERE/model" to vLLM with HF_HUB_OFFLINE=1, and the proxy calls only that localhost backend. The model/ directory contains the 4-shard Qwen2.5-VL checkpoint and its index, matching the reviewed inventories. The trajectory shows these shards are the phase-2 soup output, checked during packaging (trajectory#190, trajectory#210). The final proxy edits (trajectory#221-224) changed only latency and token limits, not the model path.
✓harness credentials used only by the harnessThe trajectory shows no harness-credential use in participant code: no ANTHROPIC or OpenRouter key usage and no 'claude
The trajectory shows no harness-credential use in participant code: no ANTHROPIC or OpenRouter key usage and no 'claude -p' calls. Credentials appear only in allowed ways: an HF token for model and dataset downloads (trajectory#13, trajectory#14), the Spin compute/R2 tooling, and the frontier selfcheck/submit CLI (trajectory#213, trajectory#225, trajectory#233). serve_proxy.py makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed5h 26m into the runtook 9m 47s
startup 135.4 s of 300 s; failed POST /v1/chat/completions: timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out; 240 s is the self-check's own limit for the first figure, while grading charges a slow first figure against the 7200 s deadline
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 135.4 s
- ✗POST /v1/chat/completions · timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out; 240 s is the self-check's own limit for the first figure, while grading charges a slow first figure against the 7200 s deadline
- 2passed5h 48m into the runtook 9m 4s
startup 123.5 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 560 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 123.5 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\\begin{tikzpicture}[\n every left delimiter/.style={xshift=1ex},\n every right delimiter/.style={xshift=-1ex},\n]\n\\matrix (m) [matrix of nodes,\n ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 12,087 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 96.1 s (1,223.2 s scaled to 611), plus 74.8 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Insurance secured. A-soup at 63% compile (5 failures were truncation at 2560 tokens — serving allows 16384). Launching a 100-sample eval on A for a better signal, and verifying the wheels copy: