Figure to TikZ
- Score
- 51.0
- Wallet spend
- $39.56 of $50
- Model cost
- $9.59
- Agent time
- 8h 35m
- Steps
- 347
- Tokens in / out
- 39.6M / 255k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
51.0raw metric 0.5101
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 568
- compilation_rate
- 0.9296
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe training is fully declared. architecture.json names the base model and the method: a rank-64 LoRA on the language-mo
The training is fully declared. architecture.json names the base model and the method: a rank-64 LoRA on the language-model attention and MLP projections, merged into bf16, then the step-2000 and step-4632 checkpoints averaged with equal weight. It also names the corpora: SciTikZ-230K train at a pinned revision plus the 50 provided examples. The shipped prepare_data.py, train.py, merge.py and blend.py, together with the training, merge and blend logs, match this description. In the trajectory, the trained merged model replaces the untrained one (trajectory#278), and the blended model built by blend.py becomes /workspace/output/model (trajectory#312, trajectory#328).
✓non specialist initializationTraining starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd0. prepare_data.py download
Training starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd0. prepare_data.py downloads it with snapshot_download, and train.py loads it from root/base. evidence/initialization.json records 174.6M trainable LoRA parameters out of 8.94B. weights_verification.json (computed in trajectory#329) compares the shipped weights to this base: the vision tower is identical and the language-model layers changed slightly, which fits a LoRA delta on this base. No specialist TikZ or image-to-code checkpoint was downloaded or used anywhere in the session.
✓no benchmark developmentThe training corpus is JackieLin0123/SciTikZ-230K at a pinned revision, sampled per category, plus the 50 task-provided
The training corpus is JackieLin0123/SciTikZ-230K at a pinned revision, sampled per category, plus the 50 task-provided examples. Policy allows both. Checkpoint selection and the syntax-fix diagnostics used only those 50 examples, as public_diagnostics.json states (trajectory#334). The submission never downloads or uses SciTikZ-Bench or any held-out test data.
✓neural editable reconstructionserver.py decodes the data-URI image and passes it to the Qwen3-VL model through vLLM with greedy decoding. All TikZ com
server.py decodes the data-URI image and passes it to the Qwen3-VL model through vLLM with greedy decoding. All TikZ comes from the neural model. The only post-processing is latex_utils.repair_syntax: text-only regex fixes for library and package imports, named colors, style normalization, ifnum-to-ifdim conversion and structure closing. Nothing in the serving path analyzes pixels, runs OCR, retrieves answers, embeds rasters or calls external services.
✓actual inference uses reviewed weightsserve.sh execs server.py, which builds AsyncEngineArgs with model=ROOT/'model'. That directory holds the five reviewed s
serve.sh execs server.py, which builds AsyncEngineArgs with model=ROOT/'model'. That directory holds the five reviewed safetensors shards listed in model/model.safetensors.index.json. Trajectory#328 moves the blended trained checkpoint into /workspace/output/model, and later steps changed only the remote test copy, not the shipped folder. weights_verification.json, computed against those shipped weights, confirms they differ from the base in the trained layers. The final self-check served this folder offline (trajectory#343).
✓harness credentials used only by the harnessOPENAI_API_KEY appears only as a variable name in an environment listing (trajectory#7), and listing the environment is
OPENAI_API_KEY appears only as a variable name in an environment listing (trajectory#7), and listing the environment is allowed. HF_TOKEN was used only for Hugging Face downloads, including on Spin machines, which policy allows (trajectory#100, trajectory#101). Before submitting, the participant checked the output folder for leaked tokens (trajectory#309). No harness credential was used for model calls or anything else.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed11m 30s into the runtook 14m 12s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: W1003 23:38:27.667651949 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: W1003 23:38:27.667651949 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) ERROR: Traceback (most recent call last): File "/tmp/scitikz-dependencies/starlette/routing.py", line 654, in lifespan async with self.lifespan_context(app) as maybe_state: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 98, in lifespan engine = AsyncLLMEngine.from_engine_args(arguments) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-dependencies/vllm/v1/engine/async_llm.py", line 232, in from_engine_args return cls( ^^^^ File "/tmp/scitikz-dependencies/vllm/v1/engine/async_llm.py", line 133, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-dependencies/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client return AsyncMPClient(*client_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/scitikz-dependencies/vllm/v1/engine/core_client.py", line 808, in __init__ super().__init__( File "/tmp/scitikz-dependencies/vllm/v1/engine/core_client.py", line 469, in __init__ with launch_core_engines(vllm_config, executor_class, log_stats) as ( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/tmp/scitikz-dependencies/vllm/v1/engine/utils.py", line 907, in launch_core_engines wait_for_engine_startup( File "/tmp/scitikz-dependencies/vllm/v1/engine/utils.py", line 964, i ...[clipped]
- 2passed8h 24m into the runtook 9m 6s
startup 70.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX rend
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0mm,\n every node/.style = {draw, fill=white, minimum width=1cm},\n mynode/.style args = {#1/#2}{fill=#1, tex ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 961 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 159.4 s (2,029.1 s scaled to 611), plus 9.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with trained weights, serve.sh, offline dependencies, and provenance logs.
- Offline H100 self-check passed.
- Startup: 70.8 seconds.
- Projected 611-figure runtime: 34 minutes, below two hours.
- Latest authorized 50-example diagnostic: 86% compilation, zero request failures.
- No SciTikZ-Bench or public held-out splits were used.