Figure to TikZ
- Score
- 52.9
- Wallet spend
- $10.00 of $10
- Model cost
- $182.35
- Agent time
- 8h 2m
- Steps
- 963
- Tokens in / out
- 120.4M / 733k
- Self-checks
- 3 passed of 4 used
- Rank at $10 wallet
- 4th of 14
Score
52.9raw metric 0.5294
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 577
- compilation_rate
- 0.9444
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the full training setup: base Qwen/Qwen3-VL-8B-Instruct at a pinned revision; LoRA r16/alpha3
architecture.json declares the full training setup: base Qwen/Qwen3-VL-8B-Instruct at a pinned revision; LoRA r16/alpha32 on the language-tower attention and MLP layers; the SciTikZ-230K parquet at a pinned revision plus the 50 supplied validation pairs; a main run of 7452 steps and a finish run of 284 steps. The shipped scripts match this declaration. training/download_sources.py fetches the pinned model and dataset. training/train.py loads the base and applies PEFT. training/run_finish.py resumes the main adapter at lr 1e-5. training/merge_adapter.py merges the adapter into the base. The main and finish training logs record matching runs. The download commands and their log are retained.
✓non specialist initializationTraining starts from Qwen3VLForConditionalGeneration.from_pretrained(/workspace/model). That directory was filled by sna
Training starts from Qwen3VLForConditionalGeneration.from_pretrained(/workspace/model). That directory was filled by snapshot_download of the general-purpose Qwen3-VL-8B-Instruct at its pinned revision (trajectory#30, evidence/main.log). The adapter tensors are standard LoRA A/B matrices on the Qwen language-model layers, and the merged shards have Qwen3-VL-8B shapes. No specialist TikZ checkpoint (for example SciTikZer) was downloaded or used as the starting point; the only extra models fetched were metric models (siglip, clip, dreamsim).
✓no benchmark developmentThe training data is the SciTikZ-230K parquet at a pinned revision plus /task/public/validation.jsonl. The trusted organ
The training data is the SciTikZ-230K parquet at a pinned revision plus /task/public/validation.jsonl. The trusted organizer note says those 50 pairs come from SciTikZ-230K and are allowed for training and selection. architecture.json explicitly excludes SciTikZ-Bench. No download or use of SciTikZ-Bench appears in the scripts, manifests or trajectory.
✓neural editable reconstructionserver.py decodes the input image from its inline base64 data URL. It then has the merged Qwen3-VL model generate TikZ s
server.py decodes the input image from its inline base64 data URL. It then has the merged Qwen3-VL model generate TikZ source with vLLM AsyncLLM at temperature 0, stopping at \end{document}. If the output is truncated, it retries once with a compact prompt. It checks the source with an offline Tectonic compile and, on failure, runs one repair pass with the same model and image. The result is editable TikZ/LaTeX produced by the neural model, with no retrieval, template lookup or external API calls.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE, installs only from local wheels and starts server.py. server.py loads AsyncLLM(model=ROOT/
serve.sh sets HF_HUB_OFFLINE, installs only from local wheels and starts server.py. server.py loads AsyncLLM(model=ROOT/'model') and AutoProcessor with local_files_only. The inventoried model/model-0000X-of-00004.safetensors files are BF16 Qwen3-VL-8B shards. packaging/stage_final_weights.py checks their sha256 hashes against the merge manifest (evidence/final_model_manifest.json, produced by training/merge_and_upload.py) before staging them as the served model directory.
✓harness credentials used only by the harnessA trajectory search for ANTHROPIC, OPENAI_API_KEY, BASE_URL, openrouter and codex found no participant use of harness cr
A trajectory search for ANTHROPIC, OPENAI_API_KEY, BASE_URL, openrouter and codex found no participant use of harness credentials. The openrouter hit is only spin help text. 'codex exec' appears only in ps process output (trajectory#845). The api_key and HF_TOKEN mentions come only from a secret-scanning rg command (trajectory#550). HF_TOKEN was used only for permitted Hugging Face downloads (trajectory#30), and spin was used only for s5cmd artifact transfer.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed17m 12s into the runtook 9m 52s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: h can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#s
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: h can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) ERROR: Traceback (most recent call last): File "/tmp/scitikz-env/lib/python3.12/site-packages/starlette/routing.py", line 654, in lifespan async with self.lifespan_context(app) as maybe_state: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 31, in lifespan engine=AsyncLLM.from_engine_args(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-env/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 232, in from_engine_args return cls( ^^^^ File "/tmp/scitikz-env/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 133, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-env/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client return AsyncMPClient(*client_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-env/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 808, in __init__ super().__init__( File "/tmp/scitikz-env/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 469, in __init__ with launch_core_engines(vllm_config, executor_class, log_stats) as ( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/tmp/scitikz-env/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 907, in launch_core_engines wait_for_engine_startup( File "/tmp/scitikz-env/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 964, i ...[clipped]
- 2passed44m 55s into the runtook 11m 1s
startup 140.7 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 140.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{arrows.meta, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 5mm and 10mm,\n every node/.style = {draw, inner sep=2pt, minimum wi ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,684 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 261.5 s (3,328.2 s scaled to 611), plus 8.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed7h 15m into the runtook 16m 1s
startup 171.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 171.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\PassOptionsToPackage{dvipsnames,svgnames,x11names}{xcolor}\n\\AddToHook{package/tikz/after}{\\usetikzlibrary{arrows,arrows.meta,calc,positioning,shapes,fit,backgrounds,chains,matrix,patterns,patterns.meta,decorations.pathmorphing,decorations.p ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 2,314 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 101.7 s (1,294.4 s scaled to 611), plus 9.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed7h 33m into the runtook 11m 3s
startup 131.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 131.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\PassOptionsToPackage{dvipsnames,svgnames,x11names}{xcolor}\n\\AddToHook{package/tikz/after}{\\usetikzlibrary{arrows,arrows.meta,calc,positioning,shapes,fit,backgrounds,chains,matrix,patterns,patterns.meta,decorations.pathmorphing,decorations.p ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 2,314 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 207.1 s (2,635.7 s scaled to 611), plus 8.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the trained model and serve.sh.
- H100 check passed: 131-second startup; 44-minute projected runtime for 611 figures.
- Training-example diagnostics: 42–44 of 50 figures rendered.
- A subsequent compiler fix passed local checks on all 50 references and seven API-routing cases.