Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $7.20 of $10
- Model cost
- $5.14
- Agent time
- 2h 24m
- Steps
- 230
- Tokens in / out
- 80.9M / 372k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.3778
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startMaple starts from the pinned checkpoint. In trajectory step 41 the participant ran snapshot_download of deepgrove/maple-
Maple starts from the pinned checkpoint. In trajectory step 41 the participant ran snapshot_download of deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 into /workspace/dev/maple. tools/pack_ternary.py reads those source shards through model.safetensors.index.json and records that revision as source_revision. evidence/pack_verification.txt reports 0 mismatches across 20.2B values compared with the base weights at that revision. tools/train_connector.py loads the LM from that same directory and keeps it frozen. The packed checkpoint has exactly Maple's shapes: 24 layers, 256 experts, GQA qkv [3072], and vocab 151936x2048. No foreign tensors are present.
✓only vision encoder and connector addedThe model has three parts: the frozen Maple LM, a frozen SigLIP2 vision encoder (weights/vision/model.safetensors, visio
The model has three parts: the frozen Maple LM, a frozen SigLIP2 vision encoder (weights/vision/model.safetensors, vision_model.* tensors only, no text tower) and the connector (weights/connector.safetensors: fc1, fc2 and gain, F32). maple_vlm/model.py puts the connector's image embeddings between the vision tokens in Maple's input. maple_vlm/server.py builds nothing else: no extra LM, no adapters on the LM and no external model calls. The optional letter_bias.json is not shipped, so its bias defaults to zero.
✓main network ternaryAll main-network linear layers (q, k, v, o, expert gate, up and down) are stored as 2-bit codes, four per byte, with one
All main-network linear layers (q, k, v, o, expert gate, up and down) are stored as 2-bit codes, four per byte, with one alpha per row. maple_vlm/checkpoint.py unpacks them as (codes-1)*alpha, which gives exactly {-alpha, 0, +alpha} per row. tools/pack_ternary.py asserts check_ternary on every one of these tensors before packing. evidence/pack_verification.txt shows exact agreement with the base ternary weights. Only embeddings, norms, the router and lm_head are dense, as in the base model.
✓training data provenanceConnector training data is declared in architecture.json and evidence/data_manifest.json as train-* parquet shards of th
Connector training data is declared in architecture.json and evidence/data_manifest.json as train-* parquet shards of the public lmms-lab/LLaVA-OneVision-Data at a pinned revision. The listed subsets include textcaps, sharegpt4v, the cauldron sets, MathV360K_TQA and others. tools/fetch_train.py downloads only those train shards, and the commands are retained in evidence/download_commands.txt. The /task/public/validation.jsonl examples were used only for measurement and selection, which the policy allows. Trajectory step 59 confirms that the dev evaluation set at /workspace/data/val was built from /task/public/media and /task/public/validation.jsonl. I found no MMMU, held-out set, task-folder or benchmark validation/test downloads in the trajectory.
✓actual inference uses reviewed weightsI traced the serving path end to end. serve.sh runs `exec python3 -m maple_vlm.serve_main`. That calls server.start(root
I traced the serving path end to end. serve.sh runs `exec python3 -m maple_vlm.serve_main`. That calls server.start(root) and builds the Engine in maple_vlm/server.py. The Engine loads the reviewed weights: weights/maple_ternary.safetensors via CK.load_packed_lm, weights/vision via SiglipVisionModel and weights/connector.safetensors. All generation and letter scoring goes through model.lm.forward_embeds, with no other model or remote API. HF offline mode is set.
✓harness credentials used only by the harnessIn the trajectory, all compute went through the participant's spin compute exec, upload and download commands, and HF re
In the trajectory, all compute went through the participant's spin compute exec, upload and download commands, and HF reads were of public repos. validate.py was run against the local server without a judge base URL. I found no harness or judge API calls, no copied harness credentials and no nested agent sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 38m into the runtook 4m 16s
startup 70.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 9.8 s after a 0.2 s first question; at that rate the 900 held-out questions take 59 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed2h 14m into the runtook 4m 46s
startup 63.5 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 63.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 13.4 s after a 0.4 s first question; at that rate the 900 held-out questions take 81 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed2h 23m into the runtook 5m 43s
startup 64.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 64.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 18.8 s after a 1.6 s first question; at that rate the 900 held-out questions take 115 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. /workspace/output (81 files, 11 GB) was graded as it stood at submit time.
What was built
- Language model: Maple, frozen. Every attention and expert linear keeps its per-row
{-α, 0, +α}values. They are stored as 2-bit codes plus one bf16 scale per row, and the packed file reproduces the source checkpoint exactly (20.2 billion values, zero mismatches after treating −0.0 and +0.0 as equal). - Vision: SigLIP2 so400m-384, frozen. The 27×27 patch grid is 2×2 pooled to 196 tokens per image, placed between
<|vision_start|>and<|vision_end|>at each<image k>reference. - Connector: a two-layer MLP with RMS norm, trained for 8,442 steps (about 135k samples from public training shards of LLaVA-OneVision-Data). The language model and encoder were not updated.
- Answering: for multiple choice, the question is asked once per cyclic ordering of its options, the option probabilities are averaged, and the best option's letter is returned. Open questions get a short greedy answer. This needs no labels.
- Serving:
serve.shinstalls the vendored torch 2.8.0 (CUDA 12.8) and transformers 4.57.1 wheels into/tmpoffline, then starts an OpenAI-compatible server. I disabled cuDNN attention, which otherwise cost about 1.4 s per decode step.
Results
- Public examples (129 items, 120 multiple choice), used for measurement only:
- Text-only direct baseline: 37.5%
- Connector, single option order: 39.2%
- Connector with option-order averaging: 49.2%
- These are diagnostic numbers on the provided examples, not a held-out estimate.
- Final selfcheck (#3) passed on the submitted serving code and weights: startup 64.7 s of 300 s, projected evaluation 115 s of 3,600 s, 150/150 replies valid.
Decisions and caveats
- Chain-of-thought with a 768-token budget was tested on 5 items and abandoned: the budget ran out and answers were wrong.
- I considered a fixed letter-bias correction, but a split-half check showed no reliable gain, so it is not shipped.
- I chose the stage-4 connector over stage 3 (39.2% vs 40.8% direct), a two-item difference treated as noise.
- The architecture review did not run during selfcheck, so the evidence files are the record for it. They include the data manifest with download commands, the training logs, the packing verification, and the vision extraction checksums.
Mistake on the spin budget I extended a GPU lease by 90 minutes instead of 15 and was charged $4.31 at once. About $2.81 of the $10 wallet remained when I stopped. The lesson is recorded in memory.