Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $50.15 of $200
- Model cost
- $38.25
- Agent time
- 4h 28m
- Steps
- 558
- Tokens in / out
- 1.8M / 346k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 2nd of 19
Score
0.0raw metric 0.3533
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe architecture.json file names deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The traje
The architecture.json file names deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The trajectory backs this up: the local /artifacts/maple manifest lists that repo and revision and is marked verified (steps 1-35). On the GPU hosts, the weights were fetched with snapshot_download at the same revision into /artifacts/maple/weights (steps 57-98). train_vlm.py builds MapleVLM from those weights and trains only a delta. export_submission.py copied the base shards from /artifacts/maple/weights into output/model. It then patched only the 219 trained tensors and checked all 18651 backbone_contract keys (steps 413-417, 528).
✓only vision encoder and connector addedmodeling_maple_vlm.py adds only a SigLIP2 so400m vision encoder and a MapleVisionConnector. The connector fuses ViT laye
modeling_maple_vlm.py adds only a SigLIP2 so400m vision encoder and a MapleVisionConnector. The connector fuses ViT layers, applies a pixel-shuffle, an MLP, positional embeddings and a norm. Its outputs replace the image_pad token embeddings in Maple's own embedding sequence. model/model-vision.safetensors holds only vision_encoder.* and vision_connector.* tensors. server.py loads no second LM, no captioning front end and makes no external model call.
✓main network ternaryThe MoE experts are frozen and loaded unchanged into stacked tensors. Attention q/k/v/o is trained with a ternary STE (Q
The MoE experts are frozen and loaded unchanged into stacked tensors. Attention q/k/v/o is trained with a ternary STE (QAT) and projected to row-wise ternary {-a,0,+a} values in two places: in the exported shards, where the export verified 96 ternary attention matrices, and at serve time via ternarize_language_backbone_(). Norms and router gates are trained but stay in their original non-ternary form, as in Maple. The language network keeps its ternary weights and semantics.
✓training data provenanceTraining used only the train splits of HuggingFaceM4/the_cauldron and lmms-lab/LLaVA-OneVision-Data subsets. All of them
Training used only the train splits of HuggingFaceM4/the_cauldron and lmms-lab/LLaVA-OneVision-Data subsets. All of them are declared in architecture.json and logs/data_manifest.json, and logs/train.log matches the manifest. Checkpoint selection used /task/public/validation.jsonl, the shipped public examples, which operator policy allows. The trajectory shows no use of MMMU or frontier-ml-tasks data, and no validation or test split was loaded. Extra subsets that the code or download script can touch (UniGeo, GeoQA+, geo170k, VisualWebInstruct) are also public training data. Those last two appeared only in a later resumed run, not in the exported run1 best_delta (steps 404, 528). The retained download script and log mean these sources are recorded rather than unverified.
✓actual inference uses reviewed weightsserve.sh sets MAPLE_MODEL_DIR to the shipped model/ directory and MAPLE_VISION_DIR to vision_encoder/, then runs server.
serve.sh sets MAPLE_MODEL_DIR to the shipped model/ directory and MAPLE_VISION_DIR to vision_encoder/, then runs server.py. server.py builds MapleVLM with load_llm_weights=True. That reads model.safetensors.index.json, the patched model-0000x shards and model-vision.safetensors, then ternarizes attention. Both multiple-choice and open answers are generated by this same vlm. MAPLE_DELTA_CKPT is not set, so no other checkpoint is loaded, and I found no decoy weights.
✓harness credentials used only by the harnessThe trajectory shows judge calls used a key the participant created with 'spin openrouter key create' (step 57 onward),
The trajectory shows judge calls used a key the participant created with 'spin openrouter key create' (step 57 onward), passed to /task/validate.py (steps 337 and later). The model list came from 'spin openrouter models' (step 36). Both are the allowed Spin/OpenRouter proxy route. HF_TOKEN was read from token files only for downloads. I found no use of the harness API key or base URL, no keys echoed into files, and no extra claude, codex or gemini CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 34m into the runtook 21m 11s
startup 96.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 96.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 157.4 s after a 1.7 s first question; at that rate the 900 held-out questions take 946 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed4h 23m into the runtook 13m 29s
startup 108.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 108.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 142.0 s after a 2.0 s first question; at that rate the 900 held-out questions take 854 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests