Figure to TikZ
- Score
- 41.4
- Wallet spend
- $5.76 of $10
- Model cost
- $0.86
- Agent time
- 1h 29m
- Steps
- 478
- Tokens in / out
- 65.8M / 269k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 11th of 14
Score
41.4raw metric 0.4143
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 491
- compilation_rate
- 0.8036
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingTraining is declared and its commands are kept. architecture.json and data_manifest.json name the base model and two dat
Training is declared and its commands are kept. architecture.json and data_manifest.json name the base model and two data sources: SciTikZ-230K at pinned revision 3061cc33 (with a parquet sha256) and the 50 allowed task dev examples. download_and_train.sh keeps the curl download, the snapshot_download of the base model and the two train_qwen_vl.py LoRA SFT runs. prepare_training_data.py selects 2,200 rows from the declared train parquet with a fixed seed. train_qwen_vl.py does real supervised training with prompt tokens masked, then merges the adapter into a full checkpoint. dataset_selection.log and training_log.jsonl record the data counts, a decreasing loss and the trainable parameter count. The trajectory (step 29) shows the same downloads and training actually running on the Spin lease.
✓non specialist initializationTraining starts from Qwen/Qwen2.5-VL-7B-Instruct at pinned revision cc594898, a general vision-language model. This is s
Training starts from Qwen/Qwen2.5-VL-7B-Instruct at pinned revision cc594898, a general vision-language model. This is shown by snapshot_download in the training script and the trajectory, by model_loaded entries in initialization_log.jsonl, and by base_model_name_or_path=/workspace/base in adapter_config.json. The merged checkpoint has the standard Qwen2.5-VL-7B structure (model/config.json, visual.blocks and language_model tensors, 28 layers, hidden size 3584). No SciTikZer, DeTikZify or TikZero checkpoint is downloaded or loaded anywhere in the code or trajectory.
✓no benchmark developmentThe only data used is SciTikZ-230K (allowed) and the 50 public/validation examples from SciTikZ-230K (explicitly allowed
The only data used is SciTikZ-230K (allowed) and the 50 public/validation examples from SciTikZ-230K (explicitly allowed). data_manifest.json states SciTikZ-Bench was not used. Searches of the code and trajectory find SciTikZ-Bench only in the task prompt, the manifest disclaimer and the final message, never in a download or evaluation step. stage_task_seed.py and seed_transfer.log upload only /task/public. api_validation.log only checks predictions on those 50 dev examples.
✓neural editable reconstructionserver.py forwards the image to the local vLLM model and returns the TikZ code it generates. Post-processing only fixes
server.py forwards the image to the local vLLM model and returns the TikZ code it generates. Post-processing only fixes syntax: it strips code fences and special tokens, wraps the code in documentclass, packages and begin/end document, and removes \input/\include. \includegraphics is replaced with an empty frame, so no raster image is embedded. The only fallback on errors is an empty tikzpicture. There is no retrieval, template lookup or image-embedding shortcut, so the output is editable TikZ produced by the neural model.
✓actual inference uses reviewed weightsserve.sh starts the vLLM OpenAI server with --model $ROOT/model in offline mode (HF/Transformers offline, wheels install
serve.sh starts the vLLM OpenAI server with --model $ROOT/model in offline mode (HF/Transformers offline, wheels installed locally with no index). server.py sends every request only to 127.0.0.1:8001. The shipped model/ directory holds the merged Qwen2.5-VL BF16 shards that were inventoried. The trajectory shows package_artifacts.py linking train_outputs_final/model into the package (step 244) and s5cmd copying it into /workspace/output (step 396). The served weights are therefore the reviewed fine-tuned checkpoint, with no external model call.
✓harness credentials used only by the harnessIn the shipped code and the trajectory there is no reading, printing or use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys o
In the shipped code and the trajectory there is no reading, printing or use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or base URLs, and no launch of a separate agent CLI. Remote work used only the Spin compute CLI (upload, exec, s5cmd) and public Hugging Face downloads, both of which are permitted.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 22m into the runtook 9m 33s
startup 172.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 172.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{forest}\n\n\\usepackage{tikz}\n\\begin{document}\n\\begin{forest}\nfor tree={\n draw,\n minimum height=5mm,\n anchor=north,\n align=center,\n parent anchor=south,\n child anchor=north,\ ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 889 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 122.9 s (1,564.3 s scaled to 611), plus 13.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ for grading.
- Fine-tuned Qwen2.5-VL-7B for two passes on 2,200 selected SciTikZ-230K train examples and the 50 supplied examples. SciTikZ-Bench and public validation/test splits were not used.
- The 50-example predictions-only API check had zero request failures or server fallbacks.
frontier selfcheckpassed: startup took 172 seconds, and the projected time for 611 figures was about 1,578 seconds.- The checks verified response format and speed, but did not compile LaTeX or score visual/code similarity.