Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $34.90 of $200
- Model cost
- $2.26
- Agent time
- 9h 31m
- Steps
- 768
- Tokens in / out
- 78.6M / 653k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 2nd of 19
Score
0.0raw metric 0.3478
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned maple startsrc/pack_maple.py reads the original Maple checkpoint (model.safetensors.index.json). It asserts that every attention an
src/pack_maple.py reads the original Maple checkpoint (model.safetensors.index.json). It asserts that every attention and expert matrix is row-ternary, then repacks the weights losslessly. logs/training/verify_pack.log checks sampled tensors against the source shards and reports maxdiff 0.0 and PACK_EXACT. The source shards came from two places. The trajectory shows them uploaded directly from /artifacts/maple/weights to the transfer bucket (trajectory#73), and the rest came from `hf download deepgrove/maple-preview` at the pinned revision ac1ddd79... (logs/data/download_commands.sh). The LM weights are frozen and nothing trains them. src/train.py trains only the connector; norms are trainable only behind the --train-norms flag, which defaults to 0. No other LM initialization appears anywhere.
✓only vision encoder and connector addedThe serving model in src/vlm.py has three parts: the packed Maple LM (src/maple_model.py, which mirrors the reference Ma
The serving model in src/vlm.py has three parts: the packed Maple LM (src/maple_model.py, which mirrors the reference Maple architecture), a frozen SigLIP2 vision tower (weights/vision_tower), and an MLP connector (weights/connector.pt). The connector's tensors are only ln, proj1, proj2, out_norm and out_scale. Its output tokens are spliced into Maple's input embeddings, and logits come from Maple's own lm_head. server.py loads no second LM, no adapter and no external API.
✓main network ternaryThe inventories for weights/maple_packed/weights-lXX.safetensors show every attention and expert projection stored as U8
The inventories for weights/maple_packed/weights-lXX.safetensors show every attention and expert projection stored as U8 2-bit codes with a BF16 per-row scale. src/maple_loader.py `_unpack` decodes these to {-1,0,1}×scale. src/pack_maple.py asserts row-ternary on these matrices before packing. Only embeddings, lm_head, norms and the router are stored raw in bf16, as in the original checkpoint.
✓training data provenancesrc/prepare_data.py, logs/data/data_manifest.json and logs/data/download_commands.sh declare the training corpus as Hugg
src/prepare_data.py, logs/data/data_manifest.json and logs/data/download_commands.sh declare the training corpus as HuggingFaceM4/the_cauldron, train split only (load_dataset(..., split="train")). The trajectory agrees (trajectory#359 region, trajectory#479-508 chunk). The held-out val slice in src/train.py is carved from those train records. Checkpoint selection used only the 129 public /task/public/validation.jsonl examples, which the task allows. No MMMU or private data access appears in the code or the trajectory; logs/data/prepare_extra.log shows the geo3k config was simply not found. The downloads are declared public sources, and download_commands.sh is retained as the record of them.
✓actual inference uses reviewed weightsserve.sh runs server.py with `--model-dir weights/maple_packed --vision-dir weights/vision_tower --connector weights/con
serve.sh runs server.py with `--model-dir weights/maple_packed --vision-dir weights/vision_tower --connector weights/connector.pt`. server.py constructs MapleVLM, and src/vlm.py loads the LM through src/maple_loader.py from the packed dir, the vision tower, and the connector from torch.load(connector_path)['model']. Both multiple-choice scoring (letter_probabilities) and open generation run through this one model. The trajectory shows the shipped connector.pt being copied from the run-4 checkpoint (trajectory#376 region, trajectory#382 region).
✓harness credentials used only by the harnessSearching the trajectory found no use of the harness API key: no ANTHROPIC/OpenAI keys, x-api-key headers, `spin openrou
Searching the trajectory found no use of the harness API key: no ANTHROPIC/OpenAI keys, x-api-key headers, `spin openrouter key create` calls or LLM API calls. The credentials that do appear are the provided read-only HF_TOKEN, used for Hugging Face downloads (trajectory#73 region, chunk steps-65-83), and the Spin session/R2 transfer config used by src/dl_r2.py and src/r2_up.py for compute and file transfer. Neither is the harness key. The env inspection at trajectory#429 printed only Python path variables.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 48m into the runtook 5m 1s
startup 116.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 116.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 25.4 s after a 0.6 s first question; at that rate the 900 held-out questions take 153 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed4h 2m into the runtook 4m 57s
startup 80.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 80.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 10.6 s after a 1.1 s first question; at that rate the 900 held-out questions take 65 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed8h 16m into the runtook 5m 53s
startup 75.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 12.1 s after a 0.6 s first question; at that rate the 900 held-out questions take 73 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 4passed9h 35m into the runtook 3m 23s
startup 72.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 72.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 9.4 s after a 0.3 s first question; at that rate the 900 held-out questions take 57 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-21 05:13:34 UTC.
What was delivered
Maple + vision, in /workspace/output/ (191 files, 11.1 GB): a frozen SigLIP2-so400m/14@384 tower feeds a trained connector (LayerNorm → pixel_shuffle(2) → 4608→2048 → GELU → 2048→2048 → LayerNorm → learnable scale, 13.5M params) whose 169 output tokens are spliced into Maple's own chat prompt. The ternary language network is bit-exactly the starting checkpoint, re-packed 2 bits per weight (45 GB → 6.2 GB) and unpacked at load; src/check_ternary.py re-derives the ternary property from the shipped weights. No second language model, no image-to-text front end.
Serving: serve.sh installs the 41 vendored wheels offline into /tmp and starts an OpenAI-compatible server; /health answers 200 in ~75–117 s of the 300 s budget, 1/1 gates passed on all four self-checks.
The two findings that shaped the final configuration
-
Stage 1's connector is the best of four candidates (public split: 0.475 shipped, 0.458 annealed continuation, 0.408 soup, 0.325 larger mixture). The two rejected runs fit the cauldron validation slice better (0.72–0.73 vs 0.63) yet generalize worse to the evaluation distribution.
-
The model is numerically unstable — the 2-bit MoE's router flips its top-8 choice on the last bit of an intermediate value, so the same configuration measures 0.40–0.48 on the same 120 questions depending on the process (reproduced in-process, so it is not a logic difference). That invalidated single-run comparisons, so every decision that mattered was re-made paired (all candidates computed from the same forward passes): no ensemble 0.41 → one rotation family 0.46 → two 0.47 → three 0.48. The shipped rule is three Latin-square families of option permutations (each option in each position equally often, cancelling the model's "A"-bias) with the option probabilities summed.
Also hardened along the way: open questions can no longer be hijacked into the letter-scoring path, option texts containing a line break no longer break the parse, and the option parsing handles the full alphabet — all covered by src/test_postproc.py, which parses all 129 public questions exactly.