Figure to TikZ
- Score
- 57.0
- Wallet spend
- $49.74 of $50
- Model cost
- $248.86
- Agent time
- 11h
- Steps
- 1,265
- Tokens in / out
- 159.1M / 1.1M
- Self-checks
- 1 passed of 4 used
- Rank at $50 wallet
- 2nd of 14
Score
57.0raw metric 0.5703
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 593
- compilation_rate
- 0.9705
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe shipped code matches the declared training. architecture.json declares a LoRA fine-tune (r32, alpha64) of Qwen/Qwen3
The shipped code matches the declared training. architecture.json declares a LoRA fine-tune (r32, alpha64) of Qwen/Qwen3-VL-8B-Instruct at revision 0c351dd: phase1 ran for 2250 steps, then phase2 for 6200 steps, and checkpoint phase2-6200 was selected. Each step has matching code: training/download_base.py downloads the base at the pinned revision, and evidence/base_download.log records that download. training/train.py loads Qwen3VLForConditionalGeneration from that base, applies PEFT LoRA to the language-model linear layers with an assistant-only image-to-TikZ loss, and records the base and revision in its run config. training/merge_adapter.py merges the adapter. model/merge_manifest.json records the same base revision, rank 32, scale 2.0, the adapter sha256 and the merged q/k/v/o/gate/up/down weights.
✓non specialist initializationTraining starts from a general model, not a TikZ specialist. Both training/download_base.py and training/train.py start
Training starts from a general model, not a TikZ specialist. Both training/download_base.py and training/train.py start from the general vision-language model Qwen/Qwen3-VL-8B-Instruct at a pinned revision, and model/merge_manifest.json records that same base revision. The weight inventory shows the standard Qwen3-VL BF16 layout (lm_head [151936,4096], language_model layers, model.visual blocks). SciTikZer, DeTikZify, TikZero and their descendants are never downloaded or loaded. In the trajectory, SciTikZer appears only in the task text, and the participant looked up only the Qwen3-VL-8B-Instruct and SciTikZ-230K pages.
✓no benchmark developmentTraining and development data come only from allowed sources. training/prepare_data.py downloads SciTikZ.parquet from Ja
Training and development data come only from allowed sources. training/prepare_data.py downloads SciTikZ.parquet from JackieLin0123/SciTikZ-230K at revision 3061cc331aa6f71cc7430e0480929d0ccddade67, which is allowed, and appends the 50 allowed /task/public/validation.jsonl examples. The 88 diagnostic rows are held out from that same training corpus. evidence/data_manifest.json lists only SciTikZ-230K plus the task examples. In the trajectory, SciTikZ-Bench appears only in the task instructions and in the participant's notes saying it was not used. No step downloads or uses it.
✓neural editable reconstructionserver.py produces editable TikZ from the fine-tuned model alone. It generates greedily from the input image with vLLM,
server.py produces editable TikZ from the fine-tuned model alone. It generates greedily from the input image with vLLM, using the merged model; there is no OCR, retrieval or answer lookup. The post-processing steps cannot write figure content. latex_dependencies.py adds only \usepackage, \usetikzlibrary and xcolor option declarations. latex_compiler.py runs local tectonic and returns diagnostics only. Any repairs come from the same model, using the prompt template in repair_prompt.py.
✓actual inference uses reviewed weightsThe serving process loads the reviewed weights. serve.sh installs from the local wheels with --no-index, sets offline mo
The serving process loads the reviewed weights. serve.sh installs from the local wheels with --no-index, sets offline mode and runs server.py. server.py builds AsyncEngineArgs(model=ROOT/'model', dtype='bfloat16') and calls that one engine for both generation and repair. It makes no external API or model calls. The model directory holds the four reviewed BF16 safetensors shards. model/scored_weight_hashes.json and model/merge_manifest.json tie those shards to the merged phase2-6200 checkpoint. The rejected auxiliary models are not deployed.
✓harness credentials used only by the harnessThe trajectory shows no misuse of harness credentials. The participant used only the allowed Spin CLI (budget, compute o
The trajectory shows no misuse of harness credentials. The participant used only the allowed Spin CLI (budget, compute offers/create/exec/upload/download, s5cmd), the frontier selfcheck tool, and public Hugging Face/web lookups. There are no hits for OpenAI or Anthropic API use, no extra agent sessions such as 'codex exec', and no use of harness API keys. API_KEY and BASE_URL appear only in Spin help text. The serving path in serve.sh and server.py is offline and uses no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed8m 12s into the runtook 18m 35s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ray-2.51.1 referencing-0.37.0 regex-2026.9.29 requests-2.34.2 rich-15.0.0 rich-toolkit-0.20.5 rign
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ray-2.51.1 referencing-0.37.0 regex-2026.9.29 requests-2.34.2 rich-15.0.0 rich-toolkit-0.20.5 rignore-0.8.1 rpds-py-2026.6.3 safetensors-0.8.0 scipy-1.18.1 sentencepiece-0.2.2 sentry-sdk-2.71.0 setproctitle-1.3.8 setuptools-80.10.2 shellingham-1.5.4 six-1.17.0 sniffio-1.3.1 starlette-0.50.0 supervisor-4.3.0 sympy-1.14.0 tabulate-0.10.0 tiktoken-0.14.0 tokenizers-0.22.1 torch-2.9.0 torchaudio-2.9.0 torchvision-0.24.0 tqdm-4.70.1 transformers-4.57.3 triton-3.5.0 typer-0.27.2 typing-inspection-0.4.4 typing_extensions-4.16.0 urllib3-2.8.0 uvicorn-0.54.0 uvloop-0.23.0 vllm-0.11.2 watchfiles-1.3.0 websockets-17.2 xformers-0.0.33.post1 xgrammar-0.1.25 yarl-1.25.1 INFO: Started server process [186] INFO: Waiting for application startup. ERROR: Traceback (most recent call last): File "/tmp/scitikz-python/starlette/routing.py", line 694, in lifespan async with self.lifespan_context(app) as maybe_state: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 22, in lifespan args=AsyncEngineArgs(model=model,tokenizer=model,dtype='bfloat16',trust_remote_code=False,max_model_len=16384,max_num_seqs=24,max_num_batched_tokens=8192,gpu_memory_utilization=0.88,enforce_eager=True,disable_log_requests=True,enable_prefix_caching=False,limit_mm_per_prompt={'image':1,'video':0},mm_processor_kwargs={'min_pixels':65536,'max_pixels':786432},seed=3407) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...[clipped]
- 2failed27m 56s into the runtook 9m 44s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: does not support headdim not being a multiple of 32. [rank0]:[W1003 23:48:46.705318030 ProcessGrou
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: does not support headdim not being a multiple of 32. [rank0]:[W1003 23:48:46.705318030 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) ERROR: Traceback (most recent call last): File "/tmp/scitikz-python/starlette/routing.py", line 694, in lifespan async with self.lifespan_context(app) as maybe_state: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 23, in lifespan engine=AsyncLLMEngine.from_engine_args(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-python/vllm/v1/engine/async_llm.py", line 232, in from_engine_args return cls( ^^^^ File "/tmp/scitikz-python/vllm/v1/engine/async_llm.py", line 133, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-python/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client return AsyncMPClient(*client_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-python/vllm/v1/engine/core_client.py", line 808, in __init__ super().__init__( File "/tmp/scitikz-python/vllm/v1/engine/core_client.py", line 469, in __init__ with launch_core_engines(vllm_config, executor_class, log_stats) as ( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/tmp/scitikz-python/vllm/v1/engine/utils.py", line 907, in launch_core_engines wait_for_engine_startup( File "/tmp/scitikz-python/vllm/v1/engine/utils.py", line 964, i ...[clipped]
- 3failed49m 29s into the runtook 11m 26s
startup 166.1 s of 300 s; failed POST /v1/chat/completions: timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out; 240 s is the self-check's own limit for the first figure, while grading charges a slow first figure against the 7200 s deadline
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 166.1 s
- ✗POST /v1/chat/completions · timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out; 240 s is the self-check's own limit for the first figure, while grading charges a slow first figure against the 7200 s deadline
- 4passed10h 49m into the runtook 12m 3s
startup 215.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 215.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=2pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{chains, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0pt,\n start chain = A going below,\n base/.style = {draw, fill=#1, minimum ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,253 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 188.6 s (2,400.5 s scaled to 611), plus 4.7 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with trained weights, serve.sh, offline dependencies, and provenance.
- Final self-check passed; startup: 215 seconds.
- Compilation: 50/50 provided examples and 211/220 unused training-corpus figures.
- Full 611-request load test: 11m 24s, zero request failures.
No SciTikZ-Bench or public validation/test splits were used.