Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $8.91 of $50
- Model cost
- $38.27
- Agent time
- 3h 40m
- Steps
- 61
- Tokens in / out
- 48.8M / 134k
- Self-checks
- 1 passed of 3 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.2456
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json names the pinned base: deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. T
architecture.json names the pinned base: deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The shipped language checkpoint is maple/ (9 safetensors shards plus config.json and modeling_maple.py). Its tensor inventory matches the released Maple layout: 24 layers, 256 experts of size 512, lm_head 151936x2048, with BF16 latent weights. serve.py loads that checkpoint frozen from MAPLE_DIR and never writes language weights. No training script updates Maple parameters: train_align.py reads only the word embeddings, and train_connector.py freezes the language model. The trajectory shows the shards staged unchanged through Spin storage (step 34). Nothing points to a different initialization.
✓only vision encoder and connector addedThe serving call graph has three parts: (1) the frozen Maple causal LM, (2) a frozen SiglipVisionModel loaded from encod
The serving call graph has three parts: (1) the frozen Maple causal LM, (2) a frozen SiglipVisionModel loaded from encoder/, and (3) a VisionConnector loaded from connector.pt. connector.pt contains only connector tensors: norms, in_proj, the mid MLP, output_scale and logit_scale, about 10.7M parameters. Snapping image tokens to the nearest Maple word embedding uses the frozen Maple embedding table. No second language model or auxiliary model is loaded.
✓main network ternaryserve.py calls patch_maple_ternary on the language model. That function ternarizes every nn.Linear row-wise (threshold 0
serve.py calls patch_maple_ternary on the language model. That function ternarizes every nn.Linear row-wise (threshold 0.7·mean|w|, alpha = mean of surviving |w|), except the mlp.gate routers, which stay fp32 as the released config specifies. It covers attention projections, all expert projections and lm_head. install_grouped_moe then stacks those ternarized expert weights with the same routing math. The serving log in the trajectory reports 'ternarized 18529 language linears; grouped moe blocks 24' (step 50), which is 24 layers × 772 linears plus lm_head. Minor inaccuracy: architecture.json says lm_head stays bf16, but the code ternarizes it too; that does not break ternarity.
✓training data provenanceThe served connector was trained with InfoNCE (train_align.py) on HuggingFaceM4/the_cauldron train-split configs listed
The served connector was trained with InfoNCE (train_align.py) on HuggingFaceM4/the_cauldron train-split configs listed in data_manifest.json. download.log records each config as split=train. LLaVA-Pretrain captions were downloaded but not used. Earlier, unserved runs used the same cauldron train data. Searches found no validation or test split, no MMMU/MMMU-Pro use beyond the shipped public examples, and no frontier-ml-tasks held-out data. /task/public validation items were read only for smoke tests and response formatting, which the operator explicitly allows. Every corpus source is declared.
✓actual inference uses reviewed weightsserve.sh points MAPLE_DIR, ENCODER_DIR and CONNECTOR_PATH at the shipped maple/, encoder/ and connector.pt, sets offline
serve.sh points MAPLE_DIR, ENCODER_DIR and CONNECTOR_PATH at the shipped maple/, encoder/ and connector.pt, sets offline mode and execs serve.py. serve.py loads exactly those files, patches Maple to ternary with grouped MoE, and generates with the Maple LM on embeddings that include the snapped connector outputs. Those weights match the inventoried tensors. The trajectory self-check (step 58) passed with this serve path. Serving pools multi-image grids further, but that is a non-learned pooling step and uses the same connector weights.
✓harness credentials used only by the harnessTrajectory searches found no use, copying or echoing of harness API keys or judge base URLs, and no launch of extra harn
Trajectory searches found no use, copying or echoing of harness API keys or judge base URLs, and no launch of extra harness CLIs. The participant read Spin's own R2 storage session file (step 29) and used Spin s5cmd/compute commands (steps 30-35, 49). Those are Spin CLI operations, which the policy allows, not harness credentials. The selfcheck and submit commands are the provided harness tools.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 42m into the runtook 15m 36s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: /submission/wheels Processing ./wheels/torch-2.6.0+cu124-cp312-cp312-linux_x86_64.whl Processing .
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: /submission/wheels Processing ./wheels/torch-2.6.0+cu124-cp312-cp312-linux_x86_64.whl Processing ./wheels/transformers-4.57.1-py3-none-any.whl Processing ./wheels/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Requirement already satisfied: safetensors in /opt/venv/lib/python3.12/site-packages (0.5.3) Processing ./wheels/safetensors-0.6.2-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl Processing ./wheels/huggingface_hub-0.35.3-py3-none-any.whl Requirement already satisfied: numpy in /opt/venv/lib/python3.12/site-packages (2.2.6) Processing ./wheels/numpy-2.5.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/regex-2026.9.29-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/tqdm-4.70.1-py3-none-any.whl Processing ./wheels/packaging-26.3-py3-none-any.whl Processing ./wheels/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Processing ./wheels/filelock-4.0.9-py3-none-any.whl Processing ./wheels/requests-2.34.2-py3-none-any.whl Processing ./wheels/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl Requirement already satisfied: idna in /opt/venv/lib/python3.12/site-packages (3.20) Processing ./wheels/urllib3-2.8.0-py3-none-any.whl Requirement already satisfied: certifi in /opt/venv/lib/python3.12/site-packages (2026.7.22) Processing ./wheels/fsspec-2026.9.0-py3-none-any.whl Requirement already satisfied: typing_extensions in /opt/venv/lib/python3.12/site-packages (4.16.0) Processing ./wheels/hf_xet-1.6.0-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl INFO: pip is looking at multiple versions of torch to determine which version is compatible with other requ ...[clipped]
- 2failed3h 0m into the runtook 16m 59s
startup 114.3 s of 300 s; 1/1 gates passed; failed batch responses: 9 of 150 replies were empty, which are wrong without a judge call; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 114.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "<tool_call>"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 165.3 s after a 2.7 s first question; at that rate the 900 held-out questions take 994 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 9 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
- 3passed3h 34m into the runtook 14m 31s
startup 103.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 103.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 159.0 s after a 3.5 s first question; at that rate the 900 held-out questions take 958 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The Maple vision server is submitted. The self-check passed on the same snapshot that was handed in.
Maple’s language network stays the released ternary checkpoint. A frozen SigLIP2 vision tower and a connector trained on public training captions feed it. Each image is turned into word embeddings the language model can read, placed after the question so the 512-token window still sees both. Multiple images are packed so the question is not pushed out of that window.
The grading-style check on one H100 reported:
- Ready in 104 seconds, inside the 300-second limit.
- All 150 public replies were non-empty text, including the four-image question that previously came back as
<tool_call>. - The projected 900-question exam takes about 16 minutes, inside the one-hour limit.