Figure to TikZ
- Score
- 7.8
- Wallet spend
- $18.73 of $200
- Model cost
- $21.31
- Agent time
- 2h 58m
- Steps
- 31
- Tokens in / out
- 31.8M / 69k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 14th of 14
Score
7.8raw metric 0.07759
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 115
- compilation_rate
- 0.1882
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json names Qwen/Qwen3-VL-4B-Instruct at revision ebb281ec70b05090aa6165b016eac8ec08e71b17, and it lists the
architecture.json names Qwen/Qwen3-VL-4B-Instruct at revision ebb281ec70b05090aa6165b016eac8ec08e71b17, and it lists the training and inference evidence files. data_manifest.json declares SciTikZ-230K (train split, revision 3061cc331aa6f71cc7430e0480929d0ccddade67) with the download command; 32000 of 230909 kept rows were used. The code chain is consistent: prepare_data.py streams the pinned revision, cache_tokens.py masks labels before the assistant span, train_packed.py runs LoRA SFT (r=32, alpha=64, q/k/v/o/gate/up/down, frozen vision tower, lr 1e-4, 1 epoch), and merge_adapter.py merges the adapter. The logs agree: prepare.log reports 230909 rows kept, cache.log reports 32000 cached, and train.log shows 66M trainable params, loss falling from about 0.64 to 0.47, a saved adapter and exit 0. logs/train_status.txt says training is still in progress; this is an earlier status note and does not contradict the finished train.log.
✓non specialist initializationInitialization is the general-purpose Qwen3-VL-4B-Instruct, downloaded at a pinned revision by train/download_model.sh a
Initialization is the general-purpose Qwen3-VL-4B-Instruct, downloaded at a pinned revision by train/download_model.sh and loaded by train_packed.py. model/config.json is the standard qwen3_vl 4B architecture: hidden 2560, 36 layers, 8 KV heads, 24-layer vision tower. The weight inventory shapes match this, for example embed_tokens [151936,2560] and q_proj [4096,2560]. Neither the code nor the trajectory loads any SciTikZer, DeTikZify or TikZero checkpoint.
✓no benchmark developmentTraining data is SciTikZ-230K only, which the policy allows. data_manifest.json and prepare_data.py exclude SciTikZ-Benc
Training data is SciTikZ-230K only, which the policy allows. data_manifest.json and prepare_data.py exclude SciTikZ-Bench and the public test splits. During development the participant ran benchmark scripts and fence tests only on public dev images and predictions such as dev-0003 (allowed public validation examples) and used the harness self-check, which sends public inputs only. The only trajectory mentions of SciTikZ-Bench are the task prompt and the exclusion docstring.
✓neural editable reconstructionserver.py feeds each image plus a fixed prompt through the processor chat template into Qwen3VLForConditionalGeneration.
server.py feeds each image plus a fixed prompt through the processor chat template into Qwen3VLForConditionalGeneration.generate: greedy decoding, max 256 new tokens, a repeat/fence stop criterion. The decoded text is turned into an editable LaTeX/TikZ document by fence_latex, which strips the stray language tag, closes delimiters and environments and wraps the result in a ```latex fence. There is no retrieval, image-embedding lookup or external API. The fallback stubs are trivial constant figures used only on errors or timeouts.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$ROOT/model, enables HF offline mode and execs server.py. server.py loads Qwen3VLForConditionalG
serve.sh sets MODEL_DIR=$ROOT/model, enables HF offline mode and execs server.py. server.py loads Qwen3VLForConditionalGeneration.from_pretrained(MODEL_DIR, local_files_only=True). No adapter directory is shipped, so the optional PEFT branch does not run. The three shipped model/ safetensors shards hold full merged Qwen3-VL tensors with no lora_* keys, which is consistent with merge_adapter.py's output. The final self-check served real model-generated TikZ from this folder (trajectory#26), and the only later change was a postprocessing string fix (trajectory#28).
✓harness credentials used only by the harnessThe trajectory shows only harness CLI use: frontier selfcheck and frontier submit (trajectory#24, trajectory#26, traject
The trajectory shows only harness CLI use: frontier selfcheck and frontier submit (trajectory#24, trajectory#26, trajectory#29, trajectory#30), plus work on the assigned Spin lease. Searches for API keys and base URLs found nothing, and no harness credential was extracted or reused for any other purpose.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 46m into the runtook 16m 20s
startup 81.2 s of 300 s; 0/1 gates passed; failed 611 figures in 7200 s: projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 551.5 s share of the deadline after a 180.1 s first figure (Batch exceeded 551.483s deadline; score is ze
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 81.2 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=2pt]{standalone}\\usepackage{tikz}\\begin{document}\\begin{tikzpicture}\\draw (0,0) circle (1);\\end{tikzpicture}\\end{document}\n```"}
- ✓closed latex fence · the first closed ```latex fence holds 149 characters of LaTeX
- ✗611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 551.5 s share of the deadline after a 180.1 s first figure (Batch exceeded 551.483s deadline; score is zero)
- 2failed2h 10m into the runtook 16m 44s
startup 97.1 s of 300 s; 0/1 gates passed; failed 611 figures in 7200 s: projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 550.5 s share of the deadline after a 192.1 s first figure (Batch exceeded 550.537s deadline; score is ze
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 97.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0mm,\n every node/.style = {\n minimum width = 12mm,\n minimum height = ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,049 characters of LaTeX
- ✗611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16, a lower bound: not all 48 figures were answered within their 550.5 s share of the deadline after a 192.1 s first figure (Batch exceeded 550.537s deadline; score is zero)
- 3passed2h 28m into the runtook 14m 31s
startup 119.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 556 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 119.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=2pt]{standalone}\\usepackage{tikz}\\begin{document}\\begin{tikzpicture}\\draw (0,0) circle (1);\\end{tikzpicture}\\end{document}\n```"}
- ✓closed latex fence · the first closed ```latex fence holds 149 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 472.9 s (6,020.0 s scaled to 611), plus 120.1 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 28 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed2h 48m into the runtook 8m 41s
startup 95.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 561 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 95.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\nlatex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0mm,\n every node/.style = {\n minimum width = 12mm,\n minimum he ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 465 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 170.8 s (2,174.0 s scaled to 611), plus 54.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. The server writes TikZ from a fine-tuned Qwen3-VL-4B and returns a complete LaTeX document in a closed latex block.
The last self-check passed: startup in 95 seconds, all 48 public answers fenced, and the 611-figure deadline projected at about 2,230 seconds against the 7,200-second limit. After that check I stripped a stray latex tag the model sometimes echoes before \documentclass, so pdfLaTeX sees a document class first.