Figure to TikZ
- Score
- 48.3
- Wallet spend
- $38.39 of $200
- Model cost
- $2.65
- Agent time
- 10h 42m
- Steps
- 921
- Tokens in / out
- 92.1M / 790k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 12th of 19
Score
48.3raw metric 0.4835
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 581
- compilation_rate
- 0.9509
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingTraining is declared and matches the code. architecture.json describes a LoRA SFT (r64/alpha128) of Qwen2.5-VL-7B-Instru
Training is declared and matches the code. architecture.json describes a LoRA SFT (r64/alpha128) of Qwen2.5-VL-7B-Instruct on SciTikZ-230K, merged into a bf16 checkpoint. src/train.py implements peft LoRA with completion-only loss, mapping each image to fenced LaTeX. logs/train_args.json shows model /root/models/qwen25vl7b, output /root/runs/run3, no resume, and the train_indices split. logs/train_summary.json reports 1380 steps and a final loss of 0.301, which matches the declaration. data_manifest.json declares the source and revision of the training corpus. model/merge_info.json records that the shipped weights are the run3/final adapter merged into the base.
✓non specialist initializationThe initialization is the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898…. src/download.sh downloads i
The initialization is the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898…. src/download.sh downloads it from the Qwen HF repo. train_args.json points to the /root/models/qwen25vl7b download. model/config.json is qwen2_5_vl with _name_or_path /root/models/qwen25vl7b. The safetensors inventory has the standard BF16 Qwen2.5-VL tensor structure. In the trajectory, the only HF downloads are Qwen and SciTikZ-230K, and no SciTikZer, DeTikZify or TikZero checkpoint was fetched or loaded. One caveat: download.sh resolves main rather than the pinned revision, so the exact base revision is unverified, but the base is non-specialist either way.
✓no benchmark developmentDevelopment used only allowed data. Training and selection data come from SciTikZ-230K, which is declared in data_manife
Development used only allowed data. Training and selection data come from SciTikZ-230K, which is declared in data_manifest.json and was fetched in the trajectory. The 50 public validation examples are excluded from training by hash. The local dev set (src/make_devset.py) is 176 SciTikZ-230K rows held out from the training indices. src/after_train.sh evaluates on that dev set, and src/oracle_predictions.py only checks the scorer on the allowed public split. SciTikZ-Bench and other benchmark test data appear only as prompt or README text and were never downloaded or used.
✓neural editable reconstructionserver.py (identical to src/server.py) decodes the input image and runs greedy generation through the fine-tuned Qwen2.5
server.py (identical to src/server.py) decodes the input image and runs greedy generation through the fine-tuned Qwen2.5-VL model. The output is TikZ/LaTeX source that is only syntax- and fence-repaired by latex_repair.py. There is no retrieval, lookup table or copying of reference code at inference time. The output is editable TikZ produced by the neural model.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, points MODEL_DIR at the shipped model/ directory and runs server.
serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, points MODEL_DIR at the shipped model/ directory and runs server.py. server.py loads Qwen2_5_VLForConditionalGeneration.from_pretrained(MODEL_DIR) and calls model.generate on it. The shipped model/ shards are the reviewed BF16 Qwen2.5-VL safetensors listed in model.safetensors.index.json. merge_info.json, src/after_train.sh and src/upload_final.sh trace those shards to the run3/final merge in /root/merged. No external model or API is called during serving.
✓harness credentials used only by the harnessIn the trajectory, ANTHROPIC/OPENAI/OPENROUTER/GEMINI key and base-URL strings appear only in secret-scan greps and Spin
In the trajectory, ANTHROPIC/OPENAI/OPENROUTER/GEMINI key and base-URL strings appear only in secret-scan greps and Spin documentation or help text. There are no anthropic or openai client imports, no claude -p calls and no LLM API requests in the participant's code. The only external credential used was HF_TOKEN, for downloading SciTikZ-230K and Qwen (src/download.sh, which ships with a placeholder token). Serving runs offline.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed8h 5m into the runtook 10m 15s
startup 69.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix,arrows,positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n every node/.style={minimum size=1cm},\n mymatrix/.style={\n matrix of nodes,\n nodes in empty cell ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 934 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 309.7 s (3,942.3 s scaled to 611), plus 7.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed9h 15m into the runtook 11m 33s
startup 68.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix,arrows,positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n every node/.style={minimum size=1cm},\n mymatrix/.style={\n matrix of nodes,\n nodes in empty cell ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 934 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 322.8 s (4,108.4 s scaled to 611), plus 12.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed10h 32m into the runtook 10m 37s
startup 66.9 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 66.9 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix,arrows,positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n every node/.style={minimum size=1cm},\n mymatrix/.style={\n matrix of nodes,\n nodes in empty cell ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 934 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 319.3 s (4,064.1 s scaled to 611), plus 8.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output (133 files, 20.58 GB, digest 1553e429… — the digest the self-check verified):
What shipped
model/— Qwen2.5-VL-7B-Instruct (revcc594898…) with the LoRA adapter (r=64/α=128, vision tower frozen) merged in; trained only on SciTikZ-230K training rows (23 999-sample category-tempered mixture). No SciTikZ-Bench, no validation/test split of any public dataset, no embedded image, no pixel tracing, no OCR, no retrieval.serve.sh+server.py— offline OpenAI-style endpoint; vendored unmodified wheels installed into/tmpat startup.latex_repair.py— post-processor that turns the model's own text into one closedlatexfence holding a document pdfLaTeX accepts (preamble fix-ups, package/library loading, node-name, sqrt, self-coordinate, loop-trim and truncation repairs).
Final measurements (176 held-out training rows never selected by the mixture)
- score 0.4457, 170/176 documents compile (scorer counts 169 — one page too large to rasterise); siglip 0.800 / clip 0.790 / lpips 0.536 / ssim 0.652 / dreamsim 0.345 / ted 0.530 / crystalbleu 0.198
- the same repair applied to the reference code of those figures compiles 176/176
- 733 s for 176 figures; self-check
8db373052baapassed: startup 66.9 s of 300 s, 611 figures projected at 4073 s of 7200 s, all 48 answers well formed
Last change in this stretch: the \providecommand rule for undefined commands was removed. It had been credited with fixing rows but actually broke four of the 176 reference documents (it shadowed \ifnum, \fi, \pic, and pre-empted \newcommands of \tikzBRKt/\endangle), and it cost 0.001 of score versus the safe version while making the repair genuinely destructive. The shipped repair is the one that leaves real LaTeX intact.
Training-data provenance, download commands, and every initialization/training/measurement log are in data_manifest.json, logs/ and architecture.json.