Figure to TikZ
- Score
- 47.9
- Wallet spend
- $8.83 of $10
- Model cost
- $14.84
- Agent time
- 6h 8m
- Steps
- 273
- Tokens in / out
- 28.7M / 199k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 9th of 19
Score
47.9raw metric 0.4788
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 554
- compilation_rate
- 0.9067
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingThe training data source is declared. architecture.json and evidence/data-manifest.json name SciTikZ-230K (JackieLin0123
The training data source is declared. architecture.json and evidence/data-manifest.json name SciTikZ-230K (JackieLin0123/SciTikZ-230K, rev 3061cc33..., SciTikZ.parquet, sha256 78502d2f...). training/download_data.py downloads the dataset at that pinned revision. training/prepare_data.py filters, deduplicates and category-balances it into 50000 training rows and holds out 256 disjoint rows (confirmed by evidence/split-check.json). training/train.py runs LoRA SFT on those pairs. evidence/train-run3.log shows a complete 850-step DDP run (world_size 2, 87.3M trainable parameters) ending in a 'complete' event. The merge record and merge log tie the run3 adapter to the merged weights. The calibration and selection holdout comes from the same declared corpus.
✓non specialist initializationTraining starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd... This is shown by traini
Training starts from the general-purpose Qwen/Qwen3-VL-8B-Instruct at pinned revision 0c351dd... This is shown by training/download_model.py (snapshot_download), the base path in evidence/training-initialization.json, from_pretrained(local_files_only) followed by LoRA in training/train.py, and merge_and_unload onto the same base in training/merge_adapter.py. evidence/merge-coverage.json shows that the 750 merged tensors map one-to-one to the base tensors. The shipped config and shards have a standard Qwen3-VL structure. Nothing indicates initialization from a TikZ-specialist checkpoint such as DeTikZify or SciTikZer.
✓no benchmark developmentThe training, holdout and selection data all come from SciTikZ-230K, which the organizer allows. The holdout is a disjoi
The training, holdout and selection data all come from SciTikZ-230K, which the organizer allows. The holdout is a disjoint 256-row slice of that corpus, and split-check reports no id or code overlap. The participant scored locally on public/validation.jsonl, which the organizer note explicitly allows. In the trajectory, SciTikZ-Bench appears only in task text and plans. No download of benchmark test data and no training on benchmark data was found.
✓neural editable reconstructionserver/app.py runs the fine-tuned Qwen3-VL model through vLLM. It decodes the request image and generates TikZ code from
server/app.py runs the fine-tuned Qwen3-VL model through vLLM. It decodes the request image and generates TikZ code from image plus prompt with greedy sampling. Post-processing only extracts fenced code, repairs truncated documents, or returns a generic fallback document. The server has no lookup tables, retrieval or external API calls, and its output is editable TikZ source.
✓actual inference uses reviewed weightsserve.sh runs the server offline (HF offline flags set, HF_TOKEN unset) with wheels installed locally, then executes ser
serve.sh runs the server offline (HF offline flags set, HF_TOKEN unset) with wheels installed locally, then executes server/app.py. That file loads LLM(model=ROOT/'model'), the shipped directory whose 5 bf16 safetensors shards appear in the weight inventories and in model/weights-manifest.json. At trajectory#247 the merged model was downloaded from the S3 merged-model prefix, verified against weights-manifest.json sha256 values, and installed into output/model. The temporary base-model fallback was removed from output/model before this. Merged-model holdout outputs differed from the base model's outputs (trajectory#249), which is consistent with the trained weights being served.
✓harness credentials used only by the harnessNo step in the trajectory uses harness model credentials (ANTHROPIC or OPENAI keys, or a harness base URL) for the parti
No step in the trajectory uses harness model credentials (ANTHROPIC or OPENAI keys, or a harness base URL) for the participant's own purposes. The only credentials in use were HF_TOKEN for downloads, which is allowed, and Spin CLI or OpenRouter proxy listing, which is allowed. serve.sh unsets HF_TOKEN and runs fully offline.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed38m 9s into the runtook 22m 18s
startup 122.4 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 122.4 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[tikz,border=3.14mm]{standalone}\n\\usetikzlibrary{positioning,calc}\n\\begin{document}\n\\begin{tikzpicture}[node distance=0.5cm,font=\\sffamily]\n\\node[draw,fill=red!20] (A) {ABCC};\n\\node[draw,fill=blue!20,below=of A] (B) {XY ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,231 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 505.3 s (6,432.4 s scaled to 611), plus 7.0 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
- 2passed5h 22m into the runtook 12m 33s
startup 141.3 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 141.3 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{matrix, positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n mynode/.style={draw, fill=gray!30},\n mynode/.append style={#1},\n mymatrix/.style={matrix of nodes, no ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 1,231 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 395.3 s (5,032.0 s scaled to 611), plus 8.4 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
The required public validator completed successfully as a diagnostic:
- Unofficial diagnostic score: 0.473249
- Compilation: 42/50
- Request failures: 0/50
- Closed LaTeX fences: 50/50
- Responses over 64,000 characters: 0/50
No changes were made from public results. I’m running the final hash audit, then submitting the exact package that passed the H100 self-check.