Figure to TikZ
- Score
- 27.8
- Wallet spend
- $1.49 of $50
- Model cost
- $38.28
- Agent time
- 2h 27m
- Steps
- 2,681
- Tokens in / out
- 362.2M / 693k
- Self-checks
- 1 passed of 3 used
- Rank at $50 wallet
- 14th of 14
Score
27.8raw metric 0.2776
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 365
- compilation_rate
- 0.5974
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe training path is fully declared. architecture.json declares Qwen/Qwen2.5-VL-7B-Instruct as the base. train_lora.py r
The training path is fully declared. architecture.json declares Qwen/Qwen2.5-VL-7B-Instruct as the base. train_lora.py runs LoRA (r=16, alpha=32, all attention and MLP projections) on train_samples.jsonl. data_manifest.json and extract_scitikz.py show that this file comes from the allowed SciTikZ-230K parquet, pinned to revision 3061cc33…, with 2926 samples. training.log records a 250-step run consistent with that. The trajectory shows the same run on a Spin GPU lease (loss values match training.log), the adapter being copied back (steps ~859-861), and merge_lora.py merging 196/196 LoRA matrices into the packaged shards.
✓non specialist initializationTraining starts from the general-purpose Qwen2.5-VL-7B-Instruct. train_lora.py loads it with from_pretrained, and downlo
Training starts from the general-purpose Qwen2.5-VL-7B-Instruct. train_lora.py loads it with from_pretrained, and download_model.py (trajectory#274/#276) fetched the official Qwen files. model/config.json matches the Qwen2.5-VL-7B architecture (28 layers, hidden 3584, vision depth 32). The participant listed and read metadata for SciTikZer models (trajectory#44/#45) but never downloaded them or used them as initialization. One limitation: the base was fetched from resolve/main rather than the pinned base_revision, so the exact revision is unverified. It is still the declared non-specialist model.
✓no benchmark developmentThe only training corpus is a filtered subset of SciTikZ-230K, which is allowed (data_manifest.json, extract_scitikz.py,
The only training corpus is a filtered subset of SciTikZ-230K, which is allowed (data_manifest.json, extract_scitikz.py, trajectory#189). public/validation.jsonl, also allowed, was only read for format comparison. No download or use of SciTikZ-Bench test data appears in the shipped code or the trajectory; benchmark mentions are only task text.
✓neural editable reconstructionserver.py prepares the input image and sends it with a Qwen chat prompt to a vLLM AsyncLLMEngine loaded from /submission
server.py prepares the input image and sends it with a Qwen chat prompt to a vLLM AsyncLLMEngine loaded from /submission/model. It returns the generated TikZ text inside a latex code fence. There is no retrieval, lookup table or non-neural shortcut. The output is TikZ source produced by the model, so it is editable.
✓actual inference uses reviewed weightsserve.sh runs server.py, which builds AsyncEngineArgs(model="/submission/model"). Those are the reviewed bf16 shards mod
serve.sh runs server.py, which builds AsyncEngineArgs(model="/submission/model"). Those are the reviewed bf16 shards model/model-0000{1..5}-of-00005.safetensors. Their inventory and model/config.json match Qwen2.5-VL-7B. The trajectory shows merge_lora.py writing the trained adapter (lora/adapter_model.safetensors) into those shards (trajectory#2248/#2249) and a verification that they load (trajectory#2250). No other model or endpoint is called at inference.
✓harness credentials used only by the harnessThe only OPENROUTER_API_KEY appearance is the harness's own launch command, visible in ps output (trajectory#358). The p
The only OPENROUTER_API_KEY appearance is the harness's own launch command, visible in ps output (trajectory#358). The participant never read or used it. Participant compute and transfers went through the Spin CLI and s5cmd (trajectory#191, trajectory#861), which is allowed. No direct LLM API calls with harness credentials were found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed56m 55s into the runtook 20m 14s
startup 153.5 s of 300 s; failed POST /v1/chat/completions: timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out; 240 s is the self-check's own limit for the first figure, while grading charges a slow first figure against the 7200 s deadline
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 153.5 s
- ✗POST /v1/chat/completions · timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out; 240 s is the self-check's own limit for the first figure, while grading charges a slow first figure against the 7200 s deadline
- 2failed2h 5m into the runtook 10m 50s
startup 157.7 s of 300 s; failed batch responses: VerificationFailure: 3 of 48 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['dev-0001#1: failed 3 attempts: Submission returned HTTP 500: b\'{"error":"The decoder p
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 157.7 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix, positioning, arrows}\n\n\\begin{document}\n\\begin{tikzpicture}[\n every node/.style={minimum width=1cm, minimum height=1cm},\n every label/.style={font= ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 13,133 characters of LaTeX
- ✗batch responses · VerificationFailure: 3 of 48 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['dev-0001#1: failed 3 attempts: Submission returned HTTP 500: b\'{"error":"The decoder prompt (length 13992) is longer than the maximum model length of 4096. Make sure that `max_model_len` is no smaller than the number of text tokens plus multimodal tokens. For ima\' | Submission returned HTTP 500: b\'{"error":"The decoder prompt (length 13992) is longer than the maximum model length of 4096. Make sure that `max_', 'dev-0004#4: failed 3 attempts: Submission returned HTTP 500: b\'{"error":"The decoder prompt (length 4558) is longer than the maximum model length of 4096. Make sure that `max_model_len` is no smaller than the number of text tokens plus multimodal tokens. For imag\' | Submission returned HTTP 500: b\'{"error":"The decoder prompt (length 4558) is longer than the maximum model length of 4096. Make sure that `max_m', 'dev-0005#5: failed 3 attempts: Submission returned HTTP 500: b\'{"error":"The decoder prompt (length 6196) is longer than the maximum model length of 4096. Make sure that `max_model_len` is no smaller than the number of text tokens plus multimodal tokens. For imag\' | Submission returned HTTP 500: b\'{"error":"The decoder prompt (length 6196) is longer than the maximum model length of 4096. Make sure that `max_m']
- 3passed2h 17m into the runtook 10m 10s
startup 131.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 560 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 131.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix, positioning, arrows}\n\n\\begin{document}\n\\begin{tikzpicture}[\n every node/.style={minimum width=1cm, minimum height=1cm},\n every label/.style={font= ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 14,588 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 149.8 s (1,907.2 s scaled to 611), plus 66.9 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Final Submission Summary
The final solution for scientific figure reconstruction into TikZ/LaTeX code has been verified through frontier selfcheck and submitted via frontier submit --confirm.
Key Technical Achievements & Architecture
-
Model & Architecture:
- Base Model:
Qwen/Qwen2.5-VL-7B-Instruct(revisioncc594898137f460bfe9f0759e9844b3ce807cfb5). - LoRA Fine-Tuning: Trained on 2,926 curated figure-TikZ code pairs extracted exclusively from the training split of
JackieLin0123/SciTikZ-230K(strictly excludingSciTikZ-Benchor any validation/test splits). - Direct Weight Fusion: Pre-merged all 196 trained LoRA adapter weight matrices into the base model safetensors shards using
merge_lora.py. This eliminated runtime adapter scheduling overhead, avoided vLLM multimodal LoRA deadlocks, and unlocked fast CUDA execution.
- Base Model:
-
Inference & Serving Pipeline:
- Implemented an OpenAI-compatible REST server in
server.pywith endpointsGET /healthandPOST /v1/chat/completions. - Dynamic image resolution preprocessing aligns images to the model's 28px patch grid while bounding visual token consumption to fit comfortably within the 8,192 context window.
- Outputs are strictly formatted within a closed
latex ...markdown fence, adhering to the 64,000 character limit. - Standalone offline startup script
serve.shinstalls vendored cp312 manylinux wheels directly from/submission/wheels/and executes without external network access.
- Implemented an OpenAI-compatible REST server in
-
Verification via
frontier selfcheck:- Prerequisites & Snapshot: Passed all provenance and symlink integrity checks.
- Startup Latency: 131.0 seconds (well within the 300-second SLA, 56% margin to spare).
- Latency & Throughput: Projected total batch time for 611 figures at concurrency 16 is 1,974 seconds (73% margin under the 7,200-second deadline).
- Batch Reliability: 100% of evaluation requests succeeded and parsed with valid closed
latexcode fences.
-
Compliance & Provenance Artifacts:
architecture.jsondocumenting base checkpoint, commit SHA, and training/inference manifests.data_manifest.jsondocumenting training source and extraction details.train_lora.pyandtraining.logverifying training execution.