Figure to TikZ
- Score
- 46.8
- Wallet spend
- $48.86 of $50
- Model cost
- $6.74
- Agent time
- 3h 56m
- Steps
- 158
- Tokens in / out
- 14.5M / 169k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 8th of 14
Score
46.8raw metric 0.4681
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 527
- compilation_rate
- 0.8625
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json names base_model Qwen/Qwen2.5-VL-7B-Instruct at the fixed revision cc594898137f460bfe9f0759e9844b3ce80
architecture.json names base_model Qwen/Qwen2.5-VL-7B-Instruct at the fixed revision cc594898137f460bfe9f0759e9844b3ce807cfb5. The training code is kept: src/train_qwen.py does LoRA SFT (r=64) that trains the model to write the fenced TikZ/LaTeX document from the input image, src/extract_data.py prepares the SciTikZ-230K data, and src/merge_lora.py merges the adapter into the base. logs/train.log shows 1600 optimizer steps on 77,129 dataset rows, with loss falling from about 0.43 to about 0.28. logs/train_args.json points at the downloaded Qwen directory, and logs/merge.log shows the merged 4-shard output that was packaged under model/. The trajectory shows torchrun being launched on the lease (trajectory#73-91 region) and the merged weights being pulled into /workspace/output/model (trajectory#106-123 region).
✓non specialist initializationThe starting weights are the general-purpose VLM Qwen2.5-VL-7B-Instruct at a pinned revision. src/lease_bootstrap.sh and
The starting weights are the general-purpose VLM Qwen2.5-VL-7B-Instruct at a pinned revision. src/lease_bootstrap.sh and logs/hf_download.log show huggingface-cli downloading exactly that repo and revision (5 shards). train_qwen.py and merge_lora.py both load from that local directory. model/config.json and the tensor inventory match the standard Qwen2.5-VL-7B architecture (hidden size 3584, 28 layers, vocab 152064). Searching the trajectory for SciTikZer or DeTikZify found only quotes of the task rules and manifest text, no download or use of a specialist checkpoint.
✓no benchmark developmentTraining data is JackieLin0123/SciTikZ-230K at revision 3061cc33…, downloaded with the curl command kept in src/lease_bo
Training data is JackieLin0123/SciTikZ-230K at revision 3061cc33…, downloaded with the curl command kept in src/lease_bootstrap.sh and recorded in DATA_MANIFEST.md. That dataset is explicitly allowed. Evaluation used only the 50 public validation examples (organizer-confirmed as drawn from SciTikZ-230K, allowed) and a held-out split carved from SciTikZ-230K by src/extract_data.py (src/lease_eval.sh, src/lease_eval2.sh). Every SciTikZ-Bench mention in the trajectory is a quote of the rules or the manifest. Nothing shows it being downloaded or used.
✓neural editable reconstructionserve.sh runs vLLM's OpenAI-compatible server on the fine-tuned Qwen2.5-VL vision-language model. The model is trained t
serve.sh runs vLLM's OpenAI-compatible server on the fine-tuned Qwen2.5-VL vision-language model. The model is trained to take the image and generate the full LaTeX/TikZ source inside a ```latex fence (train_qwen.py). No retrieval, tracing, OCR or raster-embedding code exists anywhere in the serving path.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline (all have trusted digest audits) and then runs `vllm serve $HERE/model` wi
serve.sh installs the vendored wheels offline (all have trusted digest audits) and then runs `vllm serve $HERE/model` with HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1. model/ holds the merged 4-shard bf16 safetensors that merge_lora.py produced (sizes match logs/merge.log). Their inventory and config are consistent with Qwen2.5-VL-7B plus the merged LoRA. There are no other checkpoints, no decoy loading path, and no external service calls during serving.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENROUTER, API_KEY usage, claude CLI calls and printenv found no participant
Searches of the trajectory for ANTHROPIC, OPENROUTER, API_KEY usage, claude CLI calls and printenv found no participant use or copying of harness credentials. The only credentials the participant used were HF_TOKEN, for downloads (allowed, trajectory#53-72 region), and the Spin wallet R2 session credentials (src/s3dev.sh). The only external fetches were WebFetch reads of vLLM and transformers source on GitHub.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 31m into the runtook 21m 50s
startup 140.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 140.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix}\n\n\\begin{document}\n\\begin{tikzpicture}[every left delimiter/.style={xshift=.7em}, every right delimiter/.style={xshift=-.7em}]\n \\matrix (m) [matrix of nodes,\ ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,561 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 314.8 s (4,007.3 s scaled to 611), plus 14.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 45 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Clean — only field-name strings in the s3 helper scripts, no actual credentials. Submit: