Figure to TikZ
- Score
- 41.4
- Wallet spend
- $62.82 of $200
- Model cost
- $50.67
- Agent time
- 9h 41m
- Steps
- 419
- Tokens in / out
- 85.5M / 206k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 11th of 14
Score
41.4raw metric 0.414
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 475
- compilation_rate
- 0.7774
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json and DATA_MANIFEST.md declare the training corpus as SciTikZ-230K at revision 3061cc33, licensed Apache
architecture.json and DATA_MANIFEST.md declare the training corpus as SciTikZ-230K at revision 3061cc33, licensed Apache-2.0, and both keep the curl download command. The training was LoRA SFT on Qwen2.5-VL-7B-Instruct with r128, alpha 256 and lr 1.2e-4. It used 11,895 rows and stopped at 54% of the schedule under a 9,900 s budget. prep_data.py builds the data from the parquet only, and train.py runs the LoRA SFT and the merge. The trajectory matches these claims: #293 trims the data to 12,000 rows, #294 launches run1, #296 shows 11,895 rows remain, and #339 shows the time limit, a loss of 0.373 and the merge. logs/train_config.json and model/train_config.json both record 11,895 rows. Two minor documentation gaps do not change the declared corpus or recipe. First, the shipped logs/train.log is an earlier 30K-row log from the lium host. Second, trim_data.py was not shipped, although running it is recorded in the trajectory.
✓non specialist initializationtrain.py loads Qwen2_5_VLForConditionalGeneration from --model, which is /workspace/models/qwen25vl7b. That directory ca
train.py loads Qwen2_5_VLForConditionalGeneration from --model, which is /workspace/models/qwen25vl7b. That directory came from a snapshot_download of Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898. Training then applies a peft LoRA and calls merge_and_unload. model/config.json is a standard Qwen2.5-VL config. In the trajectory, the run is launched from the base model (#294) and merged (#339). No specialist TikZ checkpoint was used for initialization.
✓no benchmark developmentDATA_MANIFEST.md and architecture.json state that SciTikZ-Bench was never used. They also state that the 50 public dev e
DATA_MANIFEST.md and architecture.json state that SciTikZ-Bench was never used. They also state that the 50 public dev examples were used only for local scoring and model selection. Both are allowed uses. prep_data.py reads only the SciTikZ-230K parquet. The trajectory shows dev-50 used only for scoring: the base at #280, the submission at #359 and the backup at #380. No benchmark data was used for training.
✓neural editable reconstructionserver.py starts vLLM on the shipped merged Qwen2.5-VL weights and proxies /v1/chat/completions to it. Its normalize() o
server.py starts vLLM on the shipped merged Qwen2.5-VL weights and proxies /v1/chat/completions to it. Its normalize() only extracts and closes the LaTeX fence from the model's output. FALLBACK_DOC is returned only when the output is empty or an error occurs. There is no retrieval, lookup table or external API call. The output is editable TikZ/LaTeX source produced by the neural model, which the self-check sample confirms (trajectory#414).
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline, sets HF offline mode, and execs server.py. server.py launches the vLLM ap
serve.sh installs the vendored wheels offline, sets HF offline mode, and execs server.py. server.py launches the vLLM api_server with --model pointing at the bundled model directory, confirmed by the self-check log line '--model /submission/model' (trajectory#414). The shipped shards model-0000{1..4}-of-00004.safetensors and the index are the run1 merged output. The trajectory traces them from the merge (#339) through packaging (#366, #371) and a serve test (#384–392) to the pull into /workspace/output (#401–402).
✓harness credentials used only by the harnessThe only credentials used in the session were Spin-issued transfer credentials, used for S3 moves between hosts (traject
The only credentials used in the session were Spin-issued transfer credentials, used for S3 moves between hosts (trajectory#342–344). Searches of the trajectory for harness/API keys such as OPENROUTER and API_KEY, and for printenv, found no misuse. No harness credentials were used by the participant's code or commands.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed9h 18m into the runtook 21m 33s
startup 148.6 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 148.6 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=5pt]{standalone}\n\\usetikzlibrary{matrix,arrows.meta}\n\n\\begin{document}\n\\begin{tikzpicture}\n\\matrix (m) [matrix of nodes, nodes in empty cells, nodes={draw, minimum height=1cm, minimum width=1cm, outer sep=0pt ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 755 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 329.4 s (4,192.5 s scaled to 611), plus 12.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Final sanity check of evidence files and symlinks before confirming: