Figure to TikZ
- Score
- 0.0
- Wallet spend
- $47.64 of $50
- Model cost
- $3.43
- Agent time
- 11h 14m
- Steps
- 268
- Tokens in / out
- 27.5M / 62k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 19th of 19
Batch exceeded 7200s deadline; score is zero
Score
0.0
The score is the raw metric (compilation rate × geometric mean of seven figure and code similarity metrics), which already runs from 0 to 1, times 100.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓declared trainingdata_manifest.json names the training corpus: the allowed SciTikZ-230K dataset at revision 3061cc331aa6f71cc7430e0480929
data_manifest.json names the training corpus: the allowed SciTikZ-230K dataset at revision 3061cc331aa6f71cc7430e0480929d0ccddade67 (Apache-2.0), with an hf_hub_download command and 100,000 samples. The trajectory shows that download at the same revision. evidence/train_full.py runs LoRA SFT (r=64, alpha=128) on that parquet: rows with code of 4000 characters or fewer, 100k sampled with seed 42, 12,500 steps. evidence/train.log shows 100,000 training samples and the run finishing at step 12500/12500 with final_adapter saved. evidence/merge.py merges the adapter into the base model. The training data and procedure are declared and consistent.
✓non specialist initializationarchitecture.json declares the base as Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898137f460bfe9f0759e9844b3ce807cfb5.
architecture.json declares the base as Qwen/Qwen2.5-VL-7B-Instruct at revision cc594898137f460bfe9f0759e9844b3ce807cfb5. The trajectory shows that general-purpose checkpoint downloaded into /workspace/models/qwen25vl7b. train_full.py and merge.py load the base from that directory. model/model.safetensors has the Qwen2.5-VL-7B tensor structure (lm_head [152064,3584], q/k/v biases, visual.blocks) and no LoRA tensors, which is consistent with a merged Qwen base plus adapter. Searches found no use of SciTikZer, DeTikZify, TikZero or other specialist TikZ checkpoints.
✓no benchmark developmentTraining used only the SciTikZ-230K parquet. data_manifest.json lists SciTikZ-Bench as forbidden and says validation.jso
Training used only the SciTikZ-230K parquet. data_manifest.json lists SciTikZ-Bench as forbidden and says validation.jsonl was used for diagnostics only, which the policy allows. Searches of the trajectory found no SciTikZ-Bench download or use. Reading the harness evaluation.py for the request format did not involve benchmark data. The throughput test (train_bench.py) used SciTikZ-230K data. Smoke tests used a blank synthetic image.
✓neural editable reconstructionserver.py loads Qwen2_5_VLForConditionalGeneration and AutoProcessor from MODEL_DIR. For each request it decodes the ima
server.py loads Qwen2_5_VLForConditionalGeneration and AutoProcessor from MODEL_DIR. For each request it decodes the image, applies the chat template and runs greedy model.generate to produce TikZ/LaTeX text. There is no retrieval, lookup table or external API. The output is editable code generated by the neural model.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, sets MODEL_DIR to the shipped model/ directory, and runs server.p
serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, sets MODEL_DIR to the shipped model/ directory, and runs server.py. server.py loads exactly that directory, which holds the reviewed model/model.safetensors (merged Qwen2.5-VL-7B, BF16). The trajectory shows the merged model copied to output/model. Later steps changed only config.json: rope_scaling was restored from the base config. The weights were not swapped, and the server was tested successfully against the shipped model directory.
✓harness credentials used only by the harnessHF_TOKEN was used only for Hugging Face downloads: the base Qwen model and the base Qwen config and tokenizer files. The
HF_TOKEN was used only for Hugging Face downloads: the base Qwen model and the base Qwen config and tokenizer files. The policy allows this. The R2/S3 transfers used spin compute session credentials, not harness credentials. Searches found no use of harness API keys, base URLs, OpenRouter or Anthropic credentials by the participant.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed9h 40m into the runtook 17m 55s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) [1;36m(APIServer pid=186)
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained [1;36m(APIServer pid=186)[0;0m return cls._from_pretrained( [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained [1;36m(APIServer pid=186)[0;0m tokenizer = cls(*init_inputs, **init_kwargs) [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 120, in __init__ [1;36m(APIServer pid=186)[0;0m super().__init__( [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py", line 178, in __init__ [1;36m(APIServer pid=186)[0;0m super().__init__(**kwargs) [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1472, in __init__ [1;36m(APIServer pid=186)[0;0m self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens) [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens [1;36m(APIServer pid=186)[0;0m self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys()) [1;36m(APIServer pid= ...[clipped]
- 2failed10h 2m into the runtook 7m 59s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: s.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) [1;36m(APIServer pid=186)[0;
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: s.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained [1;36m(APIServer pid=186)[0;0m return cls._from_pretrained( [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2359, in _from_pretrained [1;36m(APIServer pid=186)[0;0m tokenizer = cls(*init_inputs, **init_kwargs) [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 120, in __init__ [1;36m(APIServer pid=186)[0;0m super().__init__( [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_fast.py", line 178, in __init__ [1;36m(APIServer pid=186)[0;0m super().__init__(**kwargs) [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1472, in __init__ [1;36m(APIServer pid=186)[0;0m self._set_model_specific_special_tokens(special_tokens=self.extra_special_tokens) [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1210, in _set_model_specific_special_tokens [1;36m(APIServer pid=186)[0;0m self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys()) [1;36m(APIServer pid=186) ...[clipped]
- 3failed10h 14m into the runtook 9m 17s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 6)[0;0m ^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/pyt
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 6)[0;0m ^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 203, in from_vllm_config [1;36m(APIServer pid=186)[0;0m return cls( [1;36m(APIServer pid=186)[0;0m ^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 133, in __init__ [1;36m(APIServer pid=186)[0;0m self.engine_core = EngineCoreClient.make_async_mp_client( [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client [1;36m(APIServer pid=186)[0;0m return AsyncMPClient(*client_args) [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 808, in __init__ [1;36m(APIServer pid=186)[0;0m super().__init__( [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 469, in __init__ [1;36m(APIServer pid=186)[0;0m with launch_core_engines(vllm_config, executor_class, log_stats) as ( [1;36m(APIServer pid=186)[0;0m File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ [1;36m(APIServer pid=186)[0;0m next(self.gen) [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 907, in launch_core_engines [1;36m(APIServer pid=186)[0;0m wait_for_engine_startup( [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 964, in wait_for_engine_ ...[clipped]
- 4failed10h 24m into the runtook 8m 41s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 6)[0;0m ^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/pyt
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 6)[0;0m ^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 203, in from_vllm_config [1;36m(APIServer pid=186)[0;0m return cls( [1;36m(APIServer pid=186)[0;0m ^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 133, in __init__ [1;36m(APIServer pid=186)[0;0m self.engine_core = EngineCoreClient.make_async_mp_client( [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client [1;36m(APIServer pid=186)[0;0m return AsyncMPClient(*client_args) [1;36m(APIServer pid=186)[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 808, in __init__ [1;36m(APIServer pid=186)[0;0m super().__init__( [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 469, in __init__ [1;36m(APIServer pid=186)[0;0m with launch_core_engines(vllm_config, executor_class, log_stats) as ( [1;36m(APIServer pid=186)[0;0m File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ [1;36m(APIServer pid=186)[0;0m next(self.gen) [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 907, in launch_core_engines [1;36m(APIServer pid=186)[0;0m wait_for_engine_startup( [1;36m(APIServer pid=186)[0;0m File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 964, in wait_for_engine_ ...[clipped]
The agent's last message before its session ended.
Submitted.