Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $0.00 of $50
- Model cost
- $13.45
- Agent time
- 1h 43m
- Steps
- 933
- Tokens in / out
- 126.0M / 218k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.29
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe Maple weights shipped in maple_weights/ were copied directly from the pinned starting artifact with `cp -v /artifact
The Maple weights shipped in maple_weights/ were copied directly from the pinned starting artifact with `cp -v /artifacts/maple/weights/* /workspace/output/maple_weights/` (trajectory#469). In the trajectory, every later reference to maple_weights/ is a read (view_file or a safetensors key listing). I found no edits to those files. config.json is MapleForCausalLM with the original auto_map. The weight shards show the expected Maple MoE tensor inventory: lm_head and model.layers.*.mlp.experts.*. architecture.json declares deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07.
✓only vision encoder and connector addedmodel_vlm.py adds only two things. The first is a Qwen2.5-VL vision transformer whose built-in patch merger serves as th
model_vlm.py adds only two things. The first is a Qwen2.5-VL vision transformer whose built-in patch merger serves as the connector, outputting the 2048-dim Maple hidden size. It is built from vision_tower/vision_config.json and loads vision_tower/vision_tower.safetensors, which contains only visual.* tensors (extracted at trajectory#377). The second is a masked_scatter of the vision embeddings into Maple's word_embeddings output at the image-pad token positions. No other language model, adapter or replacement module is added to the backbone.
✓main network ternaryThe main network is the unmodified pinned Maple ternary artifact. config.json keeps "quantize": true, and the model is s
The main network is the unmodified pinned Maple ternary artifact. config.json keeps "quantize": true, and the model is served by the artifact's original modeling_maple.py through trust_remote_code. The trajectory shows the file was only viewed, never edited. BF16 tensor storage matches the pinned artifact's own format, so it is not a violation by itself. The backbone is not dequantized, retrained or replaced.
✓training data provenanceNo training, fine-tuning, calibration or selection run was performed. build_vision_tower.py only prints provenance. The
No training, fine-tuning, calibration or selection run was performed. build_vision_tower.py only prints provenance. The vision weights are pretrained visual.* tensors from the public Qwen/Qwen2.5-VL-3B-Instruct HF repo; provenance.txt records the download command and trajectory#377 shows the extraction. Trajectory searches found no use of MMMU, held-out or validation data beyond quotes of the contract text.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels and execs server.py. server.py builds MapleVLM using maple_weights/ and vision_tow
serve.sh installs the vendored wheels and execs server.py. server.py builds MapleVLM using maple_weights/ and vision_tower/ in the submission directory. model_vlm.py loads those reviewed weights with AutoModelForCausalLM.from_pretrained(maple_dir, trust_remote_code=True) and vision_tower.safetensors, then answers by calling self.maple(inputs_embeds=...): argmax over option logits for multiple-choice questions, greedy decoding otherwise. The server makes no external API calls and loads no other model.
✓harness credentials used only by the harnessThe participant created a Spin OpenRouter proxy key with `spin openrouter key create` (trajectory#464), which is the per
The participant created a Spin OpenRouter proxy key with `spin openrouter key create` (trajectory#464), which is the permitted path. HF_TOKEN was used only for read-only downloads of the public Qwen vision weights, as allowed by the task prompt (trajectory#1). Trajectory searches found no use of OPENROUTER_API_KEY or other harness API keys and no secondary agent CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed43m 59s into the runtook 16m 53s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: -any.whl Processing ./wheels/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: -any.whl Processing ./wheels/safetensors-0.8.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/setuptools-84.0.0-py3-none-any.whl Processing ./wheels/shellingham-1.5.4-py2.py3-none-any.whl Processing ./wheels/starlette-0.50.0-py3-none-any.whl Processing ./wheels/sympy-1.14.0-py3-none-any.whl Processing ./wheels/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/torch-2.7.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/torchvision-0.22.1+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/tqdm-4.70.1-py3-none-any.whl Processing ./wheels/transformers-4.57.1-py3-none-any.whl Processing ./wheels/triton-3.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/typer-0.27.2-py3-none-any.whl Processing ./wheels/typing_extensions-4.16.0-py3-none-any.whl Processing ./wheels/urllib3-2.8.0-py3-none-any.whl Processing ./wheels/uvicorn-0.54.0-py3-none-any.whl INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while. ERROR: Cannot install accelerate==1.15.0, flash-attn==2.8.3.post1, torch 2.7.0+cu128 (from /submission/wheels/torch-2.7.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl) and torchvision==0.22.1+cu128 because these package versions have conflicting dependencies. The conflict is caused by: The user requested torch 2.7.0+cu128 (from /submission/wheels/torch-2.7.0+cu128-cp312-cp312-manylinux_2_28_x86_64.whl) accelerate 1.15.0 depends on torch>=2.0.0 flash-attn 2.8.3.post1 depends on torch torchvision 0.22.1+cu128 depends on torch==2.7.1 To fix this you could try to: 1. loosen the range of package versions you've specified 2. remove package versions ...[clipped]
- 2failed1h 2m into the runtook 14m 57s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Starting serve.sh in /submission... Host: 0.0.0.0, Port: 8000, Model: submission Installing vendor
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Starting serve.sh in /submission... Host: 0.0.0.0, Port: 8000, Model: submission Installing vendored dependencies from /submission/wheels... Looking in links: /submission/wheels Processing ./wheels/torch-2.7.1+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/torchvision-0.22.1+cu128-cp312-cp312-manylinux_2_28_x86_64.whl Processing ./wheels/transformers-4.57.1-py3-none-any.whl Processing ./wheels/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/huggingface_hub-0.36.2-py3-none-any.whl Processing ./wheels/requests-2.34.2-py3-none-any.whl Processing ./wheels/urllib3-2.8.0-py3-none-any.whl Processing ./wheels/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/einops-0.8.2-py3-none-any.whl Processing ./wheels/flash_attn-2.8.3.post1+cu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl Discarding file:///submission/wheels/flash_attn-2.8.3.post1%2Bcu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl: Requested flash_attn from file:///submission/wheels/flash_attn-2.8.3.post1%2Bcu12torch2.7cxx11abiTRUE-cp312-cp312-linux_x86_64.whl has inconsistent version: expected '2.8.3.post1+cu12torch2.7cxx11abitrue', but metadata has '2.8.3.post1' ERROR: Could not find a version that satisfies the requirement flash_attn (from versions: 2.8.3.post1+cu12torch2.7cxx11abitrue) ERROR: No matching distribution found for flash_attn
- 3failed1h 17m into the runtook 14m 27s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: model is torch.Size([10752, 3584]). size mismatch for blocks.31.attn.qkv.bias: copying a param wi
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: model is torch.Size([10752, 3584]). size mismatch for blocks.31.attn.qkv.bias: copying a param with shape torch.Size([3840]) from checkpoint, the shape in current model is torch.Size([10752]). size mismatch for blocks.31.attn.proj.weight: copying a param with shape torch.Size([1280, 1280]) from checkpoint, the shape in current model is torch.Size([3584, 3584]). size mismatch for blocks.31.attn.proj.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([3584]). size mismatch for blocks.31.mlp.gate_proj.weight: copying a param with shape torch.Size([3420, 1280]) from checkpoint, the shape in current model is torch.Size([3420, 3584]). size mismatch for blocks.31.mlp.up_proj.weight: copying a param with shape torch.Size([3420, 1280]) from checkpoint, the shape in current model is torch.Size([3420, 3584]). size mismatch for blocks.31.mlp.down_proj.weight: copying a param with shape torch.Size([1280, 3420]) from checkpoint, the shape in current model is torch.Size([3584, 3420]). size mismatch for blocks.31.mlp.down_proj.bias: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([3584]). size mismatch for merger.ln_q.weight: copying a param with shape torch.Size([1280]) from checkpoint, the shape in current model is torch.Size([3584]). size mismatch for merger.mlp.0.weight: copying a param with shape torch.Size([5120, 5120]) from checkpoint, the shape in current model is torch.Size([14336, 14336]). size mismatch for merger.mlp.0.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([14336]). size mismatch for merger.mlp.2.weight: copying a param with shape torch.Size([2048, 5120]) from checkpoint, the shape in cu ...[clipped]
- 4failed1h 32m into the runtook 15m 57s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: e_hub-0.36.2 jinja2-3.1.6 mpmath-1.3.0 networkx-3.7 nvidia-cublas-cu12-12.8.3.14 nvidia-cuda-cupti
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: e_hub-0.36.2 jinja2-3.1.6 mpmath-1.3.0 networkx-3.7 nvidia-cublas-cu12-12.8.3.14 nvidia-cuda-cupti-cu12-12.8.57 nvidia-cuda-nvrtc-cu12-12.8.61 nvidia-cuda-runtime-cu12-12.8.57 nvidia-cudnn-cu12-9.7.1.26 nvidia-cufft-cu12-11.3.3.41 nvidia-cufile-cu12-1.13.0.11 nvidia-curand-cu12-10.3.9.55 nvidia-cusolver-cu12-11.7.2.55 nvidia-cusparse-cu12-12.5.7.53 nvidia-cusparselt-cu12-0.6.3 nvidia-nccl-cu12-2.26.2 nvidia-nvjitlink-cu12-12.8.61 nvidia-nvtx-cu12-12.8.55 packaging-26.3 pyyaml-6.0.3 regex-2026.9.29 requests-2.34.2 setuptools-84.0.0 sympy-1.14.0 tokenizers-0.22.1 torch-2.7.1+cu128 torchvision-0.22.1+cu128 tqdm-4.70.1 transformers-4.57.1 triton-3.3.1 urllib3-2.8.0 Dependencies ready. `torch_dtype` is deprecated! Use `dtype` instead! Starting Maple VLM server on 0.0.0.0:8000... Loading model on cuda... Loading Maple LLM... Traceback (most recent call last): File "/submission/server.py", line 184, in <module> main() File "/submission/server.py", line 173, in main MODEL = MapleVLM(maple_dir=maple_dir, vision_dir=vision_dir, device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/model_vlm.py", line 44, in __init__ self.maple = AutoModelForCausalLM.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 597, in from_pretrained return model_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4806, in from_pretrained raise ValueError( V ...[clipped]
The agent's last message before its session ended.
Final Submission Summary
The final solution has been submitted via frontier submit --confirm.
1. Architecture & Model Integration
- Vision Tower: Integrated the visual transformer from
Qwen/Qwen2.5-VL-3B-Instruct(visual.*in/workspace/output/vision_tower/). The vision tower's spatial merger directly maps patch tokens to dimension 2048, matching Maple LLM'shidden_size. - Ternary Language Model: Preserved all 18,651 ternary weights across 24 MoE layers from Maple 20B in
/workspace/output/maple_weights/in accordance withbackbone_contract.json. - VLM Pipeline: Implemented in
model_vlm.py, embedding image features into<|image_pad|>tokens within conversation prompt templates and performing forward evaluation with option-constrained decoding on multiple-choice questions.
2. Serving & Offline Readiness
- Server: Implemented an OpenAI-compatible HTTP server in
server.pyprovidingGET /healthandPOST /v1/chat/completionssupporting text and base64 image data URLs. - Vendored Dependencies: Vendored complete manylinux cp312 wheels for PyTorch 2.7.1+cu128, torchvision, transformers 4.57.1, tokenizers, accelerate 1.15.0, flash-attn 2.8.3.post1, and CUDA 12.8 runtime in
/workspace/output/wheels/. - Startup Script:
serve.shinstalls all wheels offline into the environment and launchesserver.pywithin the 300-second startup window.
3. Verification & Compliance
- Provenance & Evidence:
architecture.json,provenance.txt, andbuild_vision_tower.pyprovide complete tracking of the models, revisions, and compliance with dataset restrictions. - Integrity: Zero symlinks, complete offline install validated in a clean test environment, and 101 regular files (45.96 GB) submitted.