Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $10.47 of $50
- Model cost
- $1.76
- Agent time
- 3h 14m
- Steps
- 220
- Tokens in / out
- 18.0M / 53k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 2nd of 19
Score
0.0raw metric 0.3589
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe language model starts from the pinned deepgrove/maple-preview revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. For
The language model starts from the pinned deepgrove/maple-preview revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. For training, the participant fetched that revision with snapshot_download (trajectory#50, #203). train_connector.py points MAPLE at the HF cache snapshot for that revision. The shipped weights/maple was made with `cp -a` from the provided starting artifact /workspace/artifacts/maple/weights (trajectory#140). The only file changed afterwards is fa3.py, which was swapped for an SDPA attention fallback because flash_attn was missing. The shard inventories match the original layout: 9 BF16 shards with experts and lm_head. Because the LLM was frozen, nothing was retrained.
✓only vision encoder and connector addedThe model adds two components to Maple: a SigLIP so400m-patch14-384 vision encoder (weights/siglip/model.safetensors) an
The model adds two components to Maple: a SigLIP so400m-patch14-384 vision encoder (weights/siglip/model.safetensors) and a 2-layer MLP connector. The connector inventory has only linear1 (1152→2048) and linear2 (2048→2048), with biases. MapleVLM loads only the Maple causal LM, the SigLIP vision model with its processor, and that connector. Image features replace placeholder-token embeddings. I found no second language model or external model API in the code.
✓main network ternaryThe main network is the pinned Maple checkpoint and modeling code, shipped unchanged. config.json keeps "quantize": true
The main network is the pinned Maple checkpoint and modeling code, shipped unchanged. config.json keeps "quantize": true. modeling_maple.py was copied as-is from the artifact. The training script freezes every LLM parameter and trains only the connector. No candidate step re-quantized, densified or replaced the Maple weights; the only change is the attention kernel (fa3.py → SDPA fallback), which does not touch weight precision. I found no established violation.
✓training data provenanceThe only training corpus is derek-thomas/ScienceQA, train split, restricted to samples with images (2000 steps). This is
The only training corpus is derek-thomas/ScienceQA, train split, restricted to samples with images (2000 steps). This is declared in architecture.json and visible in code/train_connector.py (load_dataset(..., split="train")) and in trajectory#97-99. logs/training.log (6218 image samples, 2000 steps) is consistent with that. Loading the dataset also generated the validation and test splits locally, but nothing used them. MMMU was not downloaded and the frontier-ml-tasks folder was not accessed. The /task/public examples were used only for smoke tests, which is allowed. A VQAv2 train-split attempt (trajectory#212) failed and contributed nothing to the submission. The public HF source is declared and the trajectory retains the download.
✓actual inference uses reviewed weightsserve.sh runs /submission/code/server.py. server.py builds MapleVLM from /submission/weights/maple, /submission/weights/
serve.sh runs /submission/code/server.py. server.py builds MapleVLM from /submission/weights/maple, /submission/weights/siglip and /submission/weights/connector.pt, and answers /v1/chat/completions through vlm.generate, which calls llm.generate(inputs_embeds=...). The shipped connector.pt (13118001 bytes) is the one trained and downloaded in trajectory#134 and copied in trajectory#140. One minor risk: the connector is loaded only if the file exists. It is shipped, so this does not matter here.
✓harness credentials used only by the harnessSearching the trajectory found harness API keys or base URLs only in the Spin documentation; the participant never used
Searching the trajectory found harness API keys or base URLs only in the Spin documentation; the participant never used them. HF_TOKEN was used only for Hugging Face downloads on the Spin machine (trajectory#50, #203, #204), which is allowed. Spin CLI use and frontier selfcheck/submit are permitted harness tools. No other harness CLIs were launched and no credentials were copied into the submission.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 14m into the runtook 8m 31sfree (stopped before any GPU time)
failed snapshot: Submission contains a dangling or looping symlink: weights/siglip/README.md -> ../../blobs/a5fd5771ea9bec56bf88604979b486b309614463
1 gates
- ✗snapshot · Submission contains a dangling or looping symlink: weights/siglip/README.md -> ../../blobs/a5fd5771ea9bec56bf88604979b486b309614463
- 2failed1h 23m into the runtook 18m 36s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: pip install failed Processing ./wheels/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: pip install failed Processing ./wheels/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/tokenizers-0.23.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/torch-2.11.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/torchvision-0.26.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/tqdm-4.70.1-py3-none-any.whl Processing ./wheels/transformers-4.57.1-py3-none-any.whl Processing ./wheels/triton-3.6.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/typing_extensions-4.16.0-py3-none-any.whl Processing ./wheels/urllib3-2.8.0-py3-none-any.whl ERROR: Cannot install filelock 3.32.3 (from /submission/wheels/filelock-3.32.3-py3-none-any.whl) and filelock 4.2.4 (from /submission/wheels/filelock-4.2.4-py3-none-any.whl) because these package versions have conflicting dependencies. The conflict is caused by: The user requested filelock 3.32.3 (from /submission/wheels/filelock-3.32.3-py3-none-any.whl) The user requested filelock 4.2.4 (from /submission/wheels/filelock-4.2.4-py3-none-any.whl) To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions to allow pip attempt to solve the dependency conflict ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
- 3failed1h 44m into the runtook 14m 58s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: pip install failed Processing ./wheels/pillow-12.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: pip install failed Processing ./wheels/pillow-12.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl Processing ./wheels/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/regex-2026.9.29-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/requests-2.34.2-py3-none-any.whl Processing ./wheels/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/setuptools-78.1.0-py3-none-any.whl Processing ./wheels/sympy-1.14.0-py3-none-any.whl Processing ./wheels/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/torch-2.11.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/torchvision-0.26.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/tqdm-4.70.1-py3-none-any.whl Processing ./wheels/transformers-4.57.1-py3-none-any.whl Processing ./wheels/triton-3.6.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/typing_extensions-4.16.0-py3-none-any.whl Processing ./wheels/urllib3-2.8.0-py3-none-any.whl INFO: pip is looking at multiple versions of accelerate to determine which version is compatible with other requirements. This could take a while. ERROR: Could not find a version that satisfies the requirement psutil (from accelerate) (from versions: none) ERROR: No matching distribution found for psutil
- 4failed1h 59m into the runtook 14m 18s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Loading model... Traceback (most recent call last): File "/submission/code/server.py", line 23,
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Loading model... Traceback (most recent call last): File "/submission/code/server.py", line 23, in <module> vlm = MapleVLM(MAPLE_DIR, SIGLIP_DIR, connector_path=CONNECTOR, device="cuda") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/code/maple_vlm.py", line 31, in __init__ self.llm = AutoModelForCausalLM.from_pretrained(maple_path, trust_remote_code=True, dtype=torch.bfloat16, device_map=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 597, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4806, in from_pretrained raise ValueError( ValueError: Using a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate`. You can install it with `pip install accelerate`
- 5failed2h 18m into the runtook 15m 29s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: pip install failed Processing ./wheels/protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl Process
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: pip install failed Processing ./wheels/protobuf-7.36.2-cp310-abi3-manylinux2014_x86_64.whl Processing ./wheels/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/regex-2026.9.29-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/requests-2.34.2-py3-none-any.whl Processing ./wheels/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/sentencepiece-0.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/setuptools-78.1.0-py3-none-any.whl Processing ./wheels/sympy-1.14.0-py3-none-any.whl Processing ./wheels/tokenizers-0.23.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/torch-2.11.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/torchvision-0.26.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/tqdm-4.70.1-py3-none-any.whl Processing ./wheels/transformers-4.57.1-py3-none-any.whl Processing ./wheels/triton-3.6.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/typing_extensions-4.16.0-py3-none-any.whl Processing ./wheels/urllib3-2.8.0-py3-none-any.whl INFO: pip is looking at multiple versions of huggingface-hub to determine which version is compatible with other requirements. This could take a while. ERROR: Could not find a version that satisfies the requirement click<9.0.0,>=8.4.2 (from huggingface-hub) (from versions: none) ERROR: No matching distribution found for click<9.0.0,>=8.4.2
The agent's last message before its session ended.
Submitted.