Figure to TikZ
- Score
- 45.7
- Wallet spend
- $34.90 of $50
- Model cost
- $0.94
- Agent time
- 4h 33m
- Steps
- 678
- Tokens in / out
- 76.3M / 220k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 9th of 14
Score
45.7raw metric 0.4574
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 535
- compilation_rate
- 0.8756
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json sets base_model to Qwen/Qwen2.5-VL-3B-Instruct and pins base_revision to 66285546d2b821cf421d4f5eb2576
architecture.json sets base_model to Qwen/Qwen2.5-VL-3B-Instruct and pins base_revision to 66285546d2b821cf421d4f5eb2576359d3770cd3. train.py runs LoRA SFT (r=32) for image-to-TikZ on 30,000 rows of the SciTikZ-230K train split plus the 50 supplied examples. training.log shows 59.8M trainable params, 3757 steps and a falling loss. training_metrics.json records 14,601 s of training and a final loss of 0.419. merge_model.py and merge.log merge the adapter into the pinned base. The trajectory shows the training run (trajectory#30-37 for setup, plus later monitoring steps) and the merged model being copied into output/model (trajectory#638-644).
✓non specialist initializationThe base is a general multimodal model (Qwen2.5-VL-3B-Instruct) at a pinned revision. The trajectory shows snapshot_down
The base is a general multimodal model (Qwen2.5-VL-3B-Instruct) at a pinned revision. The trajectory shows snapshot_download of exactly that repo and revision (trajectory#31, #34). train.py and merge_model.py load from /workspace/models/base. The checkpoint inventory (hidden 2048, intermediate 11008, 36 layers, 32-block visual tower) matches Qwen2.5-VL-3B. No SciTikZer, DeTikZify or TikZero checkpoint was downloaded or used anywhere in the trajectory.
✓no benchmark developmentThe only training data is the SciTikZ-230K parquet at revision 3061cc33..., declared in source_manifest.json, train.py a
The only training data is the SciTikZ-230K parquet at revision 3061cc33..., declared in source_manifest.json, train.py and data_download.log and downloaded at trajectory#35. The other data is the 50 organizer-supplied examples in training_data/dev_examples/validation.jsonl, which policy says come from SciTikZ-230K and may be used for training. Searches of the trajectory and code found no SciTikZ-Bench download or use. The word Bench appears only in the task text and in exclusion statements.
✓neural editable reconstructionserve_proxy.py forwards the request (image plus a system prompt) to the local vLLM server running the fine-tuned Qwen2.5
serve_proxy.py forwards the request (image plus a system prompt) to the local vLLM server running the fine-tuned Qwen2.5-VL model. It only post-processes the generated text: it extracts fences, adds a preamble and libraries, and closes open environments. There is no raster embedding, no tracing or OCR, and no stored-answer or image-keyed lookup. The only fallback is a trivial placeholder used when the engine errors.
✓actual inference uses reviewed weightsserve.sh installs wheels offline and launches vllm.entrypoints.openai.api_server with --model /submission/model in offli
serve.sh installs wheels offline and launches vllm.entrypoints.openai.api_server with --model /submission/model in offline mode. The proxy talks only to 127.0.0.1:8001. model/ holds the merged safetensors shards; their inventory matches the Qwen2.5-VL-3B architecture and has no stray adapters. merge_manifest.json names the pinned base. The trajectory shows these shards are the merge output copied from the training machine (trajectory#640-644). No external services are called during serving.
✓harness credentials used only by the harnessNo trajectory step references OPENAI_API_KEY, ANTHROPIC, any *_BASE_URL, or printenv. No extra codex, claude or other CL
No trajectory step references OPENAI_API_KEY, ANTHROPIC, any *_BASE_URL, or printenv. No extra codex, claude or other CLI sessions were launched. Compute and transfers went through the spin CLI, and HF downloads used the read-only token as permitted (trajectory#31, #35, #643).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed4h 24m into the runtook 7m 25s
startup 106.5 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 106.5 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance=0mm,\n every node/.style={draw, minimum width=1cm},\n every edge/.style={draw, -latex, thick},\n ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 822 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 160.3 s (2,040.9 s scaled to 611), plus 10.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/. It contains the merged BF16 model, offline runtime wheels, serve.sh, training code, and provenance logs.
Training used 30,000 random rows from SciTikZ-230K’s train split plus the 50 supplied examples identified as train-eligible. No SciTikZ-Bench data or held-out split was used.
frontier selfcheck passed on the submitted snapshot: startup took 106.5 seconds, all 48 responses had valid closed LaTeX fences, and the projected time for 611 figures was 2,051 seconds. Self-check did not compile the outputs. I did not separately run python /task/validate.py; the self-check’s internal server did not provide a URL for that command.
Key files: serve.sh, architecture.json, train.py, and training logs. The base was Qwen2.5-VL-3B-Instruct; training data came from SciTikZ-230K.