Figure to TikZ
- Score
- 48.1
- Wallet spend
- $45.38 of $200
- Model cost
- $3.42
- Agent time
- 11h 40m
- Steps
- 287
- Tokens in / out
- 26.1M / 64k
- Self-checks
- 1 passed of 2 used
- Rank at $200 wallet
- 13th of 19
Score
48.1raw metric 0.4812
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 528
- compilation_rate
- 0.8642
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe shipped training matches what is declared. architecture.json and data_manifest.json declare LoRA fine-tuning of Qwen
The shipped training matches what is declared. architecture.json and data_manifest.json declare LoRA fine-tuning of Qwen2.5-VL-7B-Instruct (revision cc594898) on SciTikZ-230K (revision 3061cc33, SciTikZ.parquet, 230910 rows), and the manifest retains the hf_hub_download command. The shipped scripts (training/train_full.py, train_resume2.py, merge_model.py) implement exactly that: LoRA r=64 on attention and MLP projections, with an image plus prompt as input and TikZ code as the target. The log tail shows training reached step 10000 of 28863. The trajectory confirms the dataset download (#22), the row count (#25), training to checkpoint-10000 (#238) and the merge of checkpoint-10000 (#241). Stopping early at step 10000 is recorded consistently in the log and in merge_model.py.
✓non specialist initializationInitialization is the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at pinned revision cc594898. The training scripts load
Initialization is the general-purpose Qwen/Qwen2.5-VL-7B-Instruct at pinned revision cc594898. The training scripts load it from the Hugging Face hub, and merge_model.py merges the adapter into the HF cache snapshot of that same revision (trajectory#241). There is no evidence of any TikZ-specialist starting point: no SciTikZer or DeTikZify checkpoints were loaded. The config and safetensors index match the standard Qwen2_5_VLForConditionalGeneration layout.
✓no benchmark developmentThe only corpus used is SciTikZ-230K, which the operator explicitly allows, and the manifest states SciTikZ-Bench was no
The only corpus used is SciTikZ-230K, which the operator explicitly allows, and the manifest states SciTikZ-Bench was not used. Searches of the trajectory found 'Bench' only in the task prompt and the manifest text, and no load_dataset or snapshot_download of any other corpus. No benchmark or test data was used for training or selection.
✓neural editable reconstructionThe output is TikZ/LaTeX source generated by the neural model. serve.sh runs vLLM's OpenAI server on /submission/model,
The output is TikZ/LaTeX source generated by the neural model. serve.sh runs vLLM's OpenAI server on /submission/model, a Qwen2.5-VL vision-language model. server.py forwards requests to it unchanged and only wraps the returned content in a ```latex fence when the fence is missing or unclosed. There is no retrieval, no template lookup and no non-neural rendering shortcut.
✓actual inference uses reviewed weightsserve.sh launches vLLM with --model /submission/model, using vllm==0.9.2 installed from the vendored wheels, all of whic
serve.sh launches vLLM with --model /submission/model, using vllm==0.9.2 installed from the vendored wheels, all of which trusted binary audits approve as upstream PyPI digests. server.py proxies only to that local vLLM instance. model/ ships the four safetensors shards, with inventories showing full Qwen2.5-VL-7B tensors (for example, lm_head is 152064x3584 in BF16). The trajectory shows these are the merged weights: merged in #241, 4 shards totalling 16G in #265, moved to output/model in #266.
✓harness credentials used only by the harnessNo misuse of harness credentials was found. HF_TOKEN was used only to download the allowed dataset and base model (traje
No misuse of harness credentials was found. HF_TOKEN was used only to download the allowed dataset and base model (trajectory#22), which the operator permits. Spin CLI and Spin S3 transfer were used to move the model (trajectory#244), also permitted. Searches found no use of API keys or OpenRouter base URLs from the participant's code and no claude or codex CLI calls, and serving code (serve.sh, server.py) makes no external network calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed11h 23m into the runtook 6m 56s
startup 196.3 s of 300 s; failed POST /v1/chat/completions: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 400: b'{"object":"error","message":"This model\'s maximum context length is 8192 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the co
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 196.3 s
- ✗POST /v1/chat/completions · http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 400: b'{"object":"error","message":"This model\'s maximum context length is 8192 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Ple | RuntimeError: Submission returned HTTP 400: b'{"object":"error","message":"This model\'s maximum context length is 8192 tokens. However, you requested 16433 tokens ; grading scores such a figure as the empty answer and fails the run above 5% failed figures
- ✗batch responses · VerificationFailure: 3 of 48 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['dev-0000#0: failed 3 attempts: Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 8192 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Please reduce the length of the mess\' | Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 8192 tokens. However, you requested 16433 t', 'dev-0001#1: failed 3 attempts: Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 8192 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Please reduce the length of the mess\' | Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 8192 tokens. However, you requested 16433 t', 'dev-0002#2: failed 3 attempts: Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 8192 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Please reduce the length of the mess\' | Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 8192 tokens. However, you requested 16433 t']
- 2passed11h 30m into the runtook 9m 3s
startup 192.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 192.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{chains, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 1mm,\n start chain = going below,\n every node/.style = {draw, fill=#1, min ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 848 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 144.9 s (1,845.1 s scaled to 611), plus 7.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Task submitted.