Figure to TikZ
- Score
- 30.9
- Wallet spend
- $8.24 of $10
- Model cost
- $3.10
- Agent time
- 11h 38m
- Steps
- 339
- Tokens in / out
- 25.4M / 62k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 18th of 19
Score
30.9raw metric 0.3087
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
Verifier details
- metric
- compilation_times_geomean_paper_metrics
- samples
- 611
- compiled
- 413
- compilation_rate
- 0.6759
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingarchitecture.json declares the base model, Qwen/Qwen2.5-VL-3B-Instruct, at revision 66285546. code/train_sft.py performs
architecture.json declares the base model, Qwen/Qwen2.5-VL-3B-Instruct, at revision 66285546. code/train_sft.py performs LoRA SFT (r16/alpha32 on the attention and MLP projections) on the SciTikZ-230K parquet pinned to revision 3061cc33. The data manifest and download script declare that source. evidence/train.log shows 20000 samples loaded, 37.1M trainable parameters and 2500 steps. The trajectory records the 20k training launch (#220–224) and the merge into the base model (#266). Public data source and training run are consistent across code, log and session.
✓non specialist initializationTraining starts from the general-purpose Qwen2.5-VL-3B-Instruct, downloaded in trajectory#44, loaded in train_sft.py and
Training starts from the general-purpose Qwen2.5-VL-3B-Instruct, downloaded in trajectory#44, loaded in train_sft.py and used as the merge base in trajectory#266. model/config.json and the safetensors inventory match a plain Qwen2.5-VL-3B: hidden size 2048, MLP 11008, 36 layers, merged full weights with no LoRA tensors. No SciTikZer, DeTikZify or other TikZ-specialist checkpoint was found anywhere in the code or session.
✓no benchmark developmentThe only corpus used is SciTikZ-230K at the pinned revision, which is allowed. The 50 public dev examples are allowed an
The only corpus used is SciTikZ-230K at the pinned revision, which is allowed. The 50 public dev examples are allowed and are listed in the manifest but not loaded for training. Neither the shipped files nor the trajectory show any SciTikZ-Bench data or test-set lookup.
✓neural editable reconstructionserve.sh launches a vLLM OpenAI server on the merged model directory. code/server.py only proxies chat completions to th
serve.sh launches a vLLM OpenAI server on the merged model directory. code/server.py only proxies chat completions to the local vLLM and adds or closes the ```latex fence. TikZ code is generated by the neural model; there is no retrieval, template lookup or non-neural reconstruction.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR to $DIR/model and passes it to vLLM --model. That directory holds the reviewed safetensors shard
serve.sh sets MODEL_DIR to $DIR/model and passes it to vLLM --model. That directory holds the reviewed safetensors shards (model-00001/00002 plus index), which trajectory#283 shows were extracted there from the merged checkpoint. server.py forwards only to http://127.0.0.1:8001 and makes no external model calls.
✓harness credentials used only by the harnessThe participant used only the HF_TOKEN available in the environment to download the public base model and dataset (#43–4
The participant used only the HF_TOKEN available in the environment to download the public base model and dataset (#43–44), plus Spin compute/transfer credentials, all through the harness tooling. Searches found no use of the harness API key, base URL or external model-API calls in participant code or commands.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed11h 23m into the runtook 5m 34s
startup 177.1 s of 300 s; failed POST /v1/chat/completions: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 400: b'{"object":"error","message":"This model\'s maximum context length is 16384 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the c
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 177.1 s
- ✗POST /v1/chat/completions · http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 400: b'{"object":"error","message":"This model\'s maximum context length is 16384 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Pl | RuntimeError: Submission returned HTTP 400: b'{"object":"error","message":"This model\'s maximum context length is 16384 tokens. However, you requested 16433 tokens; grading scores such a figure as the empty answer and fails the run above 5% failed figures
- ✗batch responses · VerificationFailure: 3 of 48 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['dev-0000#0: failed 3 attempts: Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 16384 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Please reduce the length of the mes\' | Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 16384 tokens. However, you requested 16433 ', 'dev-0001#1: failed 3 attempts: Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 16384 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Please reduce the length of the mes\' | Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 16384 tokens. However, you requested 16433 ', 'dev-0002#2: failed 3 attempts: Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 16384 tokens. However, you requested 16433 tokens (49 in the messages, 16384 in the completion). Please reduce the length of the mes\' | Submission returned HTTP 400: b\'{"object":"error","message":"This model\\\'s maximum context length is 16384 tokens. However, you requested 16433 ']
- 2passed11h 28m into the runtook 8m 15s
startup 207.1 s of 300 s; 1/1 gates passed; The 611-figure deadline was projected from a sample: 48 public figures at concurrency 16 against their 565 s share of it.; Nothing was compiled: answers were checked for a closed ```latex fence within 64,000 characters, not for whether pdfLaTeX ren
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 207.1 s
- ✓POST /v1/chat/completions · {"id": "dev-0000", "answer": "```latex\n\\documentclass[border=5pt]{standalone}\n\\usepackage{tikz}\n\\usetikzlibrary{positioning}\n\n\\begin{document}\n\\begin{tikzpicture}[\n node distance = 0mm,\n every node/.style = {draw, fill=#1, minimum size=2em},\n every label/.style ...[clipped]
- ✓closed latex fence · the first closed ```latex fence holds 913 characters of LaTeX
- ✓611 figures in 7200 s · projected from 48 of 611 requests at concurrency 16: 48 public figures took 150.6 s (1,917.6 s scaled to 611), plus 6.2 s for the first figure; held-out figures differ and may need longer answers
- ✓batch responses · all 48 answers parsed; 48 have a closed ```latex fence within 64,000 characters
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.