Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $14.39 of $50
- Model cost
- $7.15
- Agent time
- 3h 13m
- Steps
- 194
- Tokens in / out
- 71.2M / 352k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.3867
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe base model was downloaded at the pinned revision. training/download_commands.sh and trajectory#31-49 both run snapsh
The base model was downloaded at the pinned revision. training/download_commands.sh and trajectory#31-49 both run snapshot_download('deepgrove/maple-preview', revision='ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07') into ~/models/maple. training/train.py builds the LM with load_hf_lm_into(MAPLE_DIR). training/pack.py reads those base shards for the experts, router, norms, embeddings and head, and copies config.json and tokenizer.json from the base. The weights/lm_packed.safetensors inventory has Maple's structure: 256 experts, hidden size 2048, a 151936-entry vocabulary and QK-norm. Stage 3 starts from the stage-2 checkpoint (trajectory#156), which itself descends from that base.
✓only vision encoder and connector addedmaple_vlm/vlm.py builds MapleVLM from three parts: Maple's LM, a SigLIP2 so400m vision tower (maple_vlm/vit.py) and an M
maple_vlm/vlm.py builds MapleVLM from three parts: Maple's LM, a SigLIP2 so400m vision tower (maple_vlm/vit.py) and an MLP connector (pixel shuffle → LayerNorm → Linear → GELU → Linear). Image tokens are written into Maple's input embeddings at the IMAGE_PAD positions. The LM keeps its original architecture; only its attention weights were fine-tuned with QAT. The weights/vision_connector.safetensors inventory contains only vit.* and connector.* tensors. No other modules are added.
✓main network ternaryweights/lm_packed.safetensors stores every attention projection (q/k/v/o) and every expert matrix (w_gate/w_up/w_down) a
weights/lm_packed.safetensors stores every attention projection (q/k/v/o) and every expert matrix (w_gate/w_up/w_down) as U8 2-bit codes plus one BF16 alpha per row. maple_vlm/weights.py rebuilds each weight exactly as codes*alpha, and ternary_from_dense raises an error on any non-ternary matrix. training/pack.py ternarizes the QAT attention weights with the same quantizer as maple_vlm/lm.py. The embeddings, head, norms and router stay in higher precision, as in the base Maple model.
✓training data provenanceEvery corpus comes from the train split of a declared public dataset, listed in training/data_manifest.json, training/pr
Every corpus comes from the train split of a declared public dataset, listed in training/data_manifest.json, training/prep_data.py, training/prep_pmc.py and training/prep_textmc.py: LLaVA-ReCap-558K, the_cauldron train subsets, PMC-VQA train, SciQ/MedMCQA/ARC/OpenBookQA/AQuA train, MedQA-USMLE train and RACE train. The stage-3 replay is drawn from the stage-2 shards. The trajectory shows only file listings of validation/test splits, which is metadata (trajectory#50-63). It shows no MMMU download and no access to frontier-ml-tasks. The public validation.jsonl was used only for evaluation and selection, which the policy allows.
✓actual inference uses reviewed weightsserve.sh runs `exec python3 server.py`. server.py builds Engine(WEIGHTS) with WEIGHTS defaulting to the shipped weights/
serve.sh runs `exec python3 server.py`. server.py builds Engine(WEIGHTS) with WEIGHTS defaulting to the shipped weights/ folder. In maple_vlm/engine.py, load_packed_lm loads weights/lm_packed.safetensors and raises an error if any parameter is missing; weights/vision_connector.safetensors is loaded with strict=True. The engine makes no external API calls. Per weights/pack_info.json and trajectory#181, the shipped files are the stage-3 pack.
✓harness credentials used only by the harnessI searched the trajectory for ANTHROPIC, OPENAI, API_KEY usage, claude/codex CLI launches and model-API curl calls, and
I searched the trajectory for ANTHROPIC, OPENAI, API_KEY usage, claude/codex CLI launches and model-API curl calls, and found no violation. The curl calls only send HEAD requests to huggingface.co and pypi. The environment listing masks values (trajectory#9-20). Remote compute and storage went through the allowed Spin CLI. server.py and maple_vlm/engine.py make no external model calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 16m into the runtook 4m 18s
startup 65.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 8.8 s after a 0.3 s first question; at that rate the 900 held-out questions take 53 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed1h 23m into the runtook 6m 42s
startup 78.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 78.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 13.8 s after a 0.5 s first question; at that rate the 900 held-out questions take 84 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed2h 34m into the runtook 6m 3s
startup 65.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 9.7 s after a 0.6 s first question; at that rate the 900 held-out questions take 59 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 4passed3h 18m into the runtook 4m 9s
startup 60.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 60.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 9.6 s after a 0.4 s first question; at that rate the 900 held-out questions take 58 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output (84 files, 11.0 GB) was graded as it stood at 09:05 UTC. The final frontier selfcheck on that exact folder passed: startup 60 s of 300 s, all gates met, and a projected 58 s for the 900 held-out questions against a 3,600 s budget.
What was built
- Architecture: Maple's ternary MoE language model, unchanged in structure, with a SigLIP2 so400m vision encoder (512 px, 256 visual tokens per image) and a 2×2 pixel-shuffle MLP connector. No second language model and no captioning front end.
- Ternary language network: every attention projection and expert is stored as 2-bit codes plus one bf16 scale per row, which is exact for Maple's
{-α, 0, +α}weights. Attention was fine-tuned quantization-aware (straight-through estimator), so the forward pass stayed ternary. Embeddings, LM head, norms, and the router remain bf16/fp32, as in the base model. - Training: three stages.
- Connector alignment on LLaVA-ReCap captions (900 steps).
- Instruction tuning on training splits of The Cauldron, PMC-VQA, ScienceQA, AI2D, A-OKVQA, and text MC sets (2,800 steps).
- Continued training with MedQA-USMLE and RACE train splits plus a replay of stage-2 rows (940 steps).
- Serving:
serve.shinstalls unmodified cp312 wheels for torch 2.8.0 (CUDA 12.8) and tokenizers into/tmp, then runs a stdlib HTTP server. Replies are letter-only for multiple choice and short for open questions. Multiple-choice questions are scored over cyclic rotations of their options, which removes most position bias.
Results on the public validation examples (120 multiple choice, 9 open; my letter-match scorer, not gpt-6-luna)
- Stage 1 alone: 39/120.
- Stage 2 without rotation: 45/120. With rotation: 55/120.
- Stage 3 with rotation: 55/120. This is the submitted model.
Stage 2 and stage 3 disagree on 15 of 129 replies, 5 in each direction. I chose stage 3 because it matched stage 2 on validation and added college-level text multiple choice. That is a tie-break, not a measured gain. With 120 items the standard error is about ±4.5 points, so these numbers are indicative only, and the 9 open answers were not judged.
Data and compliance
- Only training splits were used. No MMMU or MMMU-Pro data, and no validation or test splits of any benchmark.
- VisualWebInstruct was probed and rejected: too few usable items, and it needs a 28.8 GB image archive.
- Provenance is in
training/data_manifest.json,manifest_pmc.json,manifest_textmc.json,manifest_stage3_replay.json, andtraining/download_commands.sh. Training and packing logs are intraining/logs/. architecture.jsonlists the base model and revision, the training and inference evidence files, and the training lineage.
Caveats
- Pre-trained weights and data used for training are under their own licenses; I did not audit the per-dataset licenses.
- The architecture review did not run, and it can still score the submission zero.
- I did not use reas