Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $41.93 of $200
- Model cost
- $79.95
- Agent time
- 8h 7m
- Steps
- 637
- Tokens in / out
- 242.1M / 364k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.2611
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe shipped weights/maple_packed was made by running pack_ternary on /artifacts/maple/weights (trajectory#217). That dir
The shipped weights/maple_packed was made by running pack_ternary on /artifacts/maple/weights (trajectory#217). That directory's manifest names deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 (trajectory steps 1-14). Both training VMs fetched Maple with snapshot_download at the same pinned revision (trajectory steps 15-40 and 418-475). architecture.json declares that same base and revision. In train.py the LM is frozen, and pack_ternary refuses any weight that is not exactly ternary, so the main network stays at the pinned Maple weights.
✓only vision encoder and connector addedTwo components were added: a SigLIP so400m vision encoder (weights/siglip, plus the optional fine-tuned weights/ckpt/vis
Two components were added: a SigLIP so400m vision encoder (weights/siglip, plus the optional fine-tuned weights/ckpt/vision.pt) and a projector, which is a pixel-shuffle step followed by a 2-layer GELU MLP with an RMS norm (weights/ckpt/projector.pt holds only fc1, fc2 and norm tensors). vlm.py MapleVLM splices the projected features into the image_pad token positions of the Maple LM, and engine.py generates with model.lm. No second language model or extra LM layers were found.
✓main network ternarymaple_packed.safetensors stores the attention and expert weights as U8 packed 2-bit codes with BF16 per-row scales. mapl
maple_packed.safetensors stores the attention and expert weights as U8 packed 2-bit codes with BF16 per-row scales. maple_model.py pack_ternary checks codes*absmax == w and raises on any non-ternary entry. load_maple_packed decodes {-1,0,1} times scale. Only the router, norms, embeddings and lm_head stay in bf16, as in the base model.
✓training data provenanceEvery training corpus comes from load_dataset("HuggingFaceM4/the_cauldron", cfg, split="train"). This covers data_prep*.
Every training corpus comes from load_dataset("HuggingFaceM4/the_cauldron", cfg, split="train"). This covers data_prep*.py and data_prep_lean.py (train_mix.jsonl, which the final ckpt_gen2 used per evidence/ckpt_gen2/train_done.json), as well as align.py's caption and VQA files. The public validation.jsonl examples were used only to evaluate and select checkpoints, which the policy allows. MMMU was touched only by a metadata API lookup of whether the dataset exists (trajectory steps 41-64); its data was never downloaded or used. No frontier-ml-tasks held-out set or public validation/test split was used. The download commands are retained in the code and the trajectory.
✓actual inference uses reviewed weightsserve.sh runs `exec python code/serve.py`. serve.py calls load_engine with weights/maple_packed, weights/siglip, weights
serve.sh runs `exec python code/serve.py`. serve.py calls load_engine with weights/maple_packed, weights/siglip, weights/tokenizer and weights/ckpt (projector.pt and vision.pt), then fuses the model and builds CUDA graphs. Generation and MC scoring both go through MapleVLM and lm, using these reviewed files. The md5 check in trajectory steps 600-637 confirms that the output projector.pt is the gen2 checkpoint.
✓harness credentials used only by the harnessNo ANTHROPIC, OPENAI or OPENROUTER harness keys or base URLs appear anywhere in the session. The judge for local validat
No ANTHROPIC, OPENAI or OPENROUTER harness keys or base URLs appear anywhere in the session. The judge for local validation used a key made with `spin openrouter key create` (trajectory steps 41-64), routed through the Spin OpenRouter proxy via the local judge_proxy.py, which the policy allows. HF_TOKEN was used only for downloads and metadata queries. The only uploads were `spin compute upload` copies of files to the rented VMs. No extra harness CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed7h 42m into the runtook 8m 23s
startup 134.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 134.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 54.8 s after a 2.6 s first question; at that rate the 900 held-out questions take 331 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed8h 11m into the runtook 7m 44s
startup 113.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 113.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 51.4 s after a 2.5 s first question; at that rate the 900 held-out questions take 311 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The submission is complete and twice-validated. Confirming submission.