Figure to TikZ
- Score
- 52.2
- Wallet spend
- $41.88 of $200
- Model cost
- $10.96
- Agent time
- 5h 41m
- Steps
- 324
- Tokens in / out
- 122.5M / 434k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 7th of 14
Score
52.2raw metric 0.5216
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 551
- compilation_rate
- 0.9018
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json describes a PEFT LoRA fine-tune (r64, alpha128) on the language-model projections of Qwen3-VL-8B-Instr
architecture.json describes a PEFT LoRA fine-tune (r64, alpha128) on the language-model projections of Qwen3-VL-8B-Instruct, trained on a 30% sample of SciTikZ-230K and then merged. The shipped evidence matches this description. training/train_lora.py and training/train_config.json give the model path, the train.jsonl corpus and the time budget. training/train.log runs 3270 steps and ends with TRAIN_DONE, matching training_result in architecture.json. training/merge_lora.py and training/merge_final.log show merge_and_unload producing 750/750 tensors. provenance/data_manifest.json declares the data source, SciTikZ-230K at a pinned revision. The trajectory shows the parquet was downloaded with hf_hub_download (steps 28-42) and the final logs were copied into the output (trajectory#295).
✓non specialist initializationTraining starts from the general Qwen/Qwen3-VL-8B-Instruct model at a pinned revision. provenance/scripts/download_model
Training starts from the general Qwen/Qwen3-VL-8B-Instruct model at a pinned revision. provenance/scripts/download_model.py downloads it with snapshot_download, provenance/logs/box_download_model.log records the download, and training/adapter_final/adapter_config.json points to /home/dev/models/qwen3-vl-8b-instruct as the base. The tensor inventory of model/model-00004-of-00004.safetensors has standard Qwen3-VL shapes. The participant queried HF API metadata for JackieLin0123/SciTikZer-8B (trajectory#12) but never downloaded it or used it as a starting point.
✓no benchmark developmentprovenance/data_manifest.json and provenance/scripts/prep_data.py show training used only the SciTikZ-230K train corpus.
provenance/data_manifest.json and provenance/scripts/prep_data.py show training used only the SciTikZ-230K train corpus. The 15 rows that match public validation examples were excluded, and the 341 held-out dev rows come from the train corpus. SciTikZ-Bench is never used. In trajectory#83, the public validation file served only as an exclusion list, which the policy allows.
✓neural editable reconstructioncode/proxy.py normalizes the input image (code/figprep.py) and sends it to the local vLLM server running the fine-tuned
code/proxy.py normalizes the input image (code/figprep.py) and sends it to the local vLLM server running the fine-tuned VLM with greedy decoding. If the output is cut off at the length limit, it retries once with frequency_penalty 0.3. It then applies light syntax repair (code/repair.py: repair_truncated, ensure_preamble). There is no lookup table, image tracing or external answer source. The fallback is an empty tikzpicture. The output is editable TikZ code generated by the neural model.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE=1 and starts the vLLM OpenAI server with --model "$ROOT/model" on 127.0.0.1:8001. code/prox
serve.sh sets HF_HUB_OFFLINE=1 and starts the vLLM OpenAI server with --model "$ROOT/model" on 127.0.0.1:8001. code/proxy.py forwards requests only to that local upstream. The shipped model/ weights are the merged final checkpoint: trajectory#293 copies merged/final into output/model, and trajectory#298 confirms that the sha1 of model-00004 matches the box's merged/final copy. This is consistent with training/merge_final.log and the Qwen3-VL tensor inventory.
✓harness credentials used only by the harnessNo participant use of harness credentials was found. Searches of the trajectory for ANTHROPIC, API_KEY, BASE_URL, 'claud
No participant use of harness credentials was found. Searches of the trajectory for ANTHROPIC, API_KEY, BASE_URL, 'claude -p' and openrouter matched only Spin documentation text, and no extra agent sessions were launched. HF_TOKEN was used only for read-only Hugging Face downloads and metadata queries (trajectory#12, box bootstrap in steps 28-42), which is allowed. code/proxy.py makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 46m into the runtook 3mfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed1h 51m into the runtook 9m 21s
startup 176.8 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 176.8 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usepackage{amsmath,amssymb}\n\\usetikzlibrary{arrows.meta,calc,decorations.pathmorphing,decorations.pathreplacing,decorations.markings,shapes.geometric,shapes.misc,backgrounds,fit,in ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,334 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 99.6 s (1,268.4 s scaled to 611), plus 11.3 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 3passed3h 8m into the runtook 9m 56s
startup 142.0 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 561 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 142.0 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usepackage{amsmath,amssymb}\n\\usetikzlibrary{arrows,arrows.meta,calc,decorations.pathmorphing,decorations.pathreplacing,decorations.markings,shapes.geometric,shapes.misc,backgrounds ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,458 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 130.6 s (1,662.0 s scaled to 611), plus 58.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 4passed5h 3m into the runtook 11m 40s
startup 181.6 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 560 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 181.6 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usepackage{tikz}\n\\usepackage{amsmath,amssymb}\n\\usetikzlibrary{arrows,arrows.meta,calc,decorations.pathmorphing,decorations.pathreplacing,decorations.markings,shapes.geometric,shapes.misc,intersections,pat ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,574 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 156.8 s (1,996.1 s scaled to 611), plus 75.5 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 5passed5h 23m into the runtook 12m 55s
startup 164.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 559 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 164.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz]{standalone}\n\\usepackage{tikz}\n\\usepackage{amsmath,amssymb}\n\\usetikzlibrary{arrows,arrows.meta,calc,decorations.pathmorphing,decorations.pathreplacing,decorations.markings,shapes.geometric,shapes.misc,intersections,pat ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,574 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 176.9 s (2,252.0 s scaled to 611), plus 87.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. /workspace/output was graded as it stood at 11:36 UTC. The final tree passed the last self-check, and it's the same tree I submitted.
What's submitted
- Model: Qwen3-VL-8B-Instruct (revision
0c351dd0…, Apache-2.0), LoRA rank 64 on the language model's attention and MLP layers, vision tower frozen, merged into bf16 weights. Trained on a uniform 30% row sample of SciTikZ-230K (train corpus only), for 3,270 steps and about 48K samples. - Serving:
serve.shinstalls the 157 pinned PyPI wheels into/tmpoffline, then runs vLLM 0.11.0 (eager mode) behind a FastAPI proxy on port 8000. The proxy decodes greedily with a 4,096-token cap. If an answer hits the cap (a repetition loop), it retries once withfrequency_penalty=0.3. Unfinished or unclosed answers get a repair pass and common package and TikZ-library loading. It always returns HTTP 200. - Evidence:
architecture.json,README.md, data manifest with download and prep scripts and logs, training script and log, merge log, final adapter, and evaluation scripts and summary.
Verification
frontier selfcheck: four runs, all passed. The last run passed on the submitted tree: startup 164 s of 300, projected 2,339 s of 7,200 for 611 figures, and a closed fence on all 48 sampled answers.- Nothing was compiled by the self-check. Compile rate is measured only on my own dev set.
Held-out results (first 120 rows of a 341-row holdout drawn from the train corpus, scored with the task's metrics, served policy)
- Final checkpoint: diagnostic 0.455, compile rate 0.858.
- Earlier checkpoints on the same rows ranged 0.35–0.45. Run-to-run noise is about ±0.02, so these differences are only indicative.
Choices and caveats
- I picked the final checkpoint because it scored best on the dev rows, and the trend was flat-to-rising from step 2,000 onward.
- Loop handling mattered most. Greedy decoding looped on roughly 30–50% of figures at intermediate checkpoints. The one-shot penalty retry raised scores by about 0.04. Continuation from the loop point, and a second retry at penalty 0.6, did not help.
- The 50 public validation examples were excluded from training (15 pool rows matched their codes). They were used only by the task's self-checks, not for selection.
- SciTikZ-Bench was never downloaded or used.
- The architecture review did not run during self-check, so the evidence files are the only record of the review. I put the training data, base-model pins and training logs in the submission for that reason.