Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $13.37 of $200
- Model cost
- $1.59
- Agent time
- 6h 16m
- Steps
- 729
- Tokens in / out
- 72.8M / 550k
- Self-checks
- 2 passed of 3 used
- Rank at $200 wallet
- 2nd of 19
Score
0.0raw metric 0.34
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned maple startexport_weights.py builds the bundled LM weights directly from the shards in /artifacts/maple/weights, which is the pinne
export_weights.py builds the bundled LM weights directly from the shards in /artifacts/maple/weights, which is the pinned deepgrove/maple-preview starting artifact. The trajectory (steps 238-240) shows the run: it read all 9 shards and wrote to /workspace/output/weights. The tensor names and shapes in weights/dense.safetensors match Maple's layout (24 layers, hidden size 2048, router, q/k norms). The language model was never trained further; only the connector was trained.
✓only vision encoder and connector addedOnly two components were added. One is the frozen Qwen2.5-VL-3B vision tower at the pinned revision; extract_vision.py c
Only two components were added. One is the frozen Qwen2.5-VL-3B vision tower at the pinned revision; extract_vision.py copies only the visual.* tensors, and weights/vision.safetensors contains no language-model weights. The other is a new RMSNorm-plus-MLP connector in weights/connector.safetensors. In server.py, image features go through the connector and replace the image pad-token embeddings before entering the unchanged Maple LM. No other models or modules are involved.
✓main network ternaryExpert weights are stored as 2-bit codes plus one bf16 scale per row (weights/exp_NN.safetensors). io_utils._dequant tur
Expert weights are stored as 2-bit codes plus one bf16 scale per row (weights/exp_NN.safetensors). io_utils._dequant turns them back into exactly {-a, 0, +a}. export_weights.py checks that the round-trip reproduces the original ternary values exactly. Attention, embedding, router and lm_head stay as Maple's original bf16 tensors, unchanged from the base model's own design.
✓training data provenanceThe connector was trained only on HuggingFaceM4/the_cauldron, train split, at the pinned revision. data_prep.py loads sp
The connector was trained only on HuggingFaceM4/the_cauldron, train split, at the pinned revision. data_prep.py loads split="train" and download_commands.sh records the download. The stage-2/3 manifest_mc.jsonl is a subset of that same converted corpus (trajectory step 352+). The /task/public examples were used only for serving-option and checkpoint selection, which is allowed (eval_val.py, README). I found no access to MMMU, the held-out maple-vlm task data, or any validation/test split of a public dataset in the code or the trajectory.
✓actual inference uses reviewed weightsserve.sh runs server.py with HF_HUB_OFFLINE=1. server.py's Engine loads only the bundled weights/ files: dense and exp_N
serve.sh runs server.py with HF_HUB_OFFLINE=1. server.py's Engine loads only the bundled weights/ files: dense and exp_NN via load_packed_lm, the vision tower via load_packed_vision, and connector.safetensors. Its forward pass goes through those modules. There are no external model API calls and no alternative model.
✓harness credentials used only by the harnessThe participant's code never calls a model API with the harness credentials, copies them, or starts extra harness CLI se
The participant's code never calls a model API with the harness credentials, copies them, or starts extra harness CLI sessions. The claude-router command line appears only in process listings (trajectory steps 238 and 245+). The R2 credentials came from `spin transfer credentials` (step 312) and were used only to move files through Spin storage (step 313+), which is allowed Spin usage. The only OpenRouter appearances are Spin help output.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed4h 10m into the runtook 4m 36s
startup 70.1 s of 300 s; 1/1 gates passed; failed batch responses: 6 of 150 replies were empty, which are wrong without a judge call; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 22.7 s after a 1.4 s first question; at that rate the 900 held-out questions take 137 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 6 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
- 2passed4h 16m into the runtook 4m 33s
startup 68.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 20.3 s after a 1.5 s first question; at that rate the 900 held-out questions take 123 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed5h 51m into the runtook 4m 29s
startup 60.5 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 60.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 25.2 s after a 1.4 s first question; at that rate the 900 held-out questions take 153 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. /workspace/output was accepted for grading at 2026-10-21 01:56:47 UTC.
Final package: Maple-preview (20B-A1B ternary MoE, language weights untouched and losslessly 2-bit-packed) + frozen Qwen2.5-VL-3B vision tower + a trained 16.8M-parameter MLP connector (stage-3 step 600), served offline on one H100 through serve.sh (POST /v1/chat/completions, /health). It passed frontier selfcheck (93cfabcce2c4: startup 60.5 s of 300 s, 1/1 gates, all 150 replies non-empty, projected evaluation 152.8 s of 3600), with the only subsequent edits being documentation in evidence/README.md and architecture.json. 77 files, 11.98 GB, no symlinks; measured multiple-choice accuracy on the provided public samples is 43.3% (52/120) with no use of any public validation/test split.