Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $112.83 of $200
- Model cost
- $13.77
- Agent time
- 10h 19m
- Steps
- 400
- Tokens in / out
- 47.3M / 426k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.3878
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json names the base as deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. In the
architecture.json names the base as deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. In the trajectory, pack_maple.py was run on the downloaded artifacts in /artifacts/maple/weights, and the end-of-run artifact manifest lists that same repo and revision with file hashes. pack_maple.py stops with an error if any source weight is not exactly ternary, and evidence/packing.log shows every layer (0-23) as exact_ternary. The shipped weights also match the base's structure: 24 layers, 256 experts each, with packed qkv, o_proj, gate_up and down tensors and BF16 routers. evidence/weights_audit_final.log reports that all 18,432 expert projections, the embedding, the output head and all 24 routers are unchanged from the pinned base. That log is the candidate's own output, but it agrees with the code and the inventories.
✓only vision encoder and connector addedThe only new components are the vision encoder and the connector. prepare_vision.py downloads Qwen/Qwen3-VL-2B-Instruct
The only new components are the vision encoder and the connector. prepare_vision.py downloads Qwen/Qwen3-VL-2B-Instruct at a pinned revision and keeps only the model.visual.* / visual.* keys. The vision.safetensors inventory has only blocks/merger/patch-embed keys and no language-model tensors. connector.safetensors holds a LayerNorm plus a two-layer MLP, and vision.py feeds its output straight into Maple as input embeddings. Inside Maple, the attention weights were re-trained but remain ternary, and the RMS norm vectors were adapted (norms.safetensors). Both are parameters of the existing network, not added modules. server.py loads no other model.
✓main network ternarymaple.py builds every Maple projection by unpacking 2-bit symbols with a per-row scale. In kernels.py, unpack_kernel map
maple.py builds every Maple projection by unpacking 2-bit symbols with a per-row scale. In kernels.py, unpack_kernel maps each 2-bit code ((byte>>2*(i%4))&3) to -1, 0 or +1 and multiplies by the row scale. The attention overrides in attention.safetensors and the expert tensors in the layer files are all packed U8 with BF16 scales. No *_latent full-precision copies ship, so the straight-through quantizer path in maple.py is never used when serving. norms.safetensors contains only norm vectors.
✓training data provenanceEvery training source is declared and pinned to a revision. download_data.py, evidence/data_manifest.json and evidence/d
Every training source is declared and pinned to a revision. download_data.py, evidence/data_manifest.json and evidence/download_files_manifest.json list train-split files only, and download_data.py rejects any file without 'train-' in its name. The sources are LLaVA-Pretrain, ScienceQA-IMG, ChartQA, geometry3k, A-OKVQA, DocVQA, TextVQA_train, and the ai2d/tqa/tabmwp/iconqa/plotqa/mapqa/vqarad subsets of the_cauldron. The download logs are retained (evidence/data_download.log, evidence/cauldron_download.log). The only data used for selection or annotation beyond these is the allowed /task/public/validation.jsonl, read by annotate_public.py. GPT-4.1 annotations were made through the Spin OpenRouter allowance, and annotate_training.py reads only train parquet files. Trajectory searches found no MMMU download, no access to the frontier-ml-tasks repo, and no validation or test splits.
✓actual inference uses reviewed weightsserve.sh installs dependencies from the local wheels and runs exec python server.py. server.py loads Maple from maple_we
serve.sh installs dependencies from the local wheels and runs exec python server.py. server.py loads Maple from maple_weights/ (dense plus layer files, with the attention.safetensors overrides) and copies in norms.safetensors. It also loads the tokenizer, vision_weights/vision.safetensors and connector.safetensors. answer_batch runs the vision encoder, then the connector, interleaves the result with Maple's own token embeddings, and decodes with Maple's ternary layers. There is no other model and no external API call in the serving path.
✓harness credentials used only by the harnessThe annotation scripts send requests using a credentials JSON file. In the trajectory, that file was created with `spin
The annotation scripts send requests using a credentials JSON file. In the trajectory, that file was created with `spin openrouter key create > /workspace/openrouter_key.json`, which is the permitted Spin OpenRouter allowance. Spin storage was also set up through the Spin CLI. I found no use of the harness's own API keys or base URL in participant code, and no extra harness CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed38m 10s into the runtook 13m 37s
startup 99.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 99.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 428.9 s after a 19.7 s first question; at that rate the 900 held-out questions take 2,593 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed2h 55m into the runtook 5m 20s
startup 76.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 76.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 89.4 s after a 0.9 s first question; at that rate the 900 held-out questions take 537 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed10h 23m into the runtook 5m 30s
startup 109.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 109.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 34.9 s after a 2.1 s first question; at that rate the 900 held-out questions take 212 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm.
- Vision encoder and connector added; Maple’s language core remains ternary.
- Final offline H100 self-check passed: 109.8-second startup, all 150 responses valid, projected evaluation 211.5 seconds.
- Supplied-example diagnostic: 127/129 correct—not a held-out accuracy claim.
- Weights, offline dependencies, training provenance, and audit evidence are included.