Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $17.26 of $50
- Model cost
- $1.14
- Agent time
- 3h 45m
- Steps
- 372
- Tokens in / out
- 34.5M / 314k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 2nd of 19
Score
0.0raw metric 0.3789
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-20).
✓pinned maple starttrain/pack_weights.py builds the shipped LM weights from /artifacts/maple/weights. The task text calls this the starting
train/pack_weights.py builds the shipped LM weights from /artifacts/maple/weights. The task text calls this the starting checkpoint pinned in STARTING_ARTIFACT.json (trajectory#230 runs the packer and copies config and tokenizer from it). weights/ternary.json names deepgrove/maple-preview@ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The remote training copy was downloaded at that exact revision (trajectory#32). The tensor shapes in the weight inventories match the Maple config (24 layers, 256 experts, hidden size 2048). The LM was not fine-tuned.
✓only vision encoder and connector addedThe only added parts are the Qwen2.5-VL-3B vision tower and a 34M-parameter MLP connector. The vision tower file (vit/vi
The only added parts are the Qwen2.5-VL-3B vision tower and a 34M-parameter MLP connector. The vision tower file (vit/vision.safetensors) contains only blocks, merger and patch_embed tensors; no Qwen language tensors (embed_tokens or layers.) were found. connector.safetensors contains only connector tensors. server.py builds just MapleLM, QwenViT and Connector, and puts image features at the image-pad positions. The CFG text-only pass reuses the same Maple LM. There is no second language model and no image-to-text front end.
✓main network ternaryThe attention and expert weights are shipped as I32 tensors holding packed 2-bit codes, plus a BF16 alpha for each row.
The attention and expert weights are shipped as I32 tensors holding packed 2-bit codes, plus a BF16 alpha for each row. maple_lm.py _load_packed unpacks them as ((p>>shifts)&3)-1 times alpha, which gives values in {-α,0,+α}. train/pack_weights.py produces these codes by thresholding each row at 0.7·mean|w|. Embeddings, norms and routers stay in higher precision, and none of them is a ternary-projection layer.
✓training data provenanceThe connector was trained on train subsets of LLaVA-OneVision-Data, downloaded with train/dl_data.sh (log in logs/dl_lov
The connector was trained on train subsets of LLaVA-OneVision-Data, downloaded with train/dl_data.sh (log in logs/dl_lovd.log) and converted by train/prep_data.py. Selection used the shipped /task/public/validation.jsonl, which is allowed. data_manifest.md and architecture.json declare all sources. Trajectory searches found no MMMU download, no use of the frontier-ml-tasks repo and no validation or test split of a public dataset.
✓actual inference uses reviewed weightsserve.sh runs server.py with MAPLE_WEIGHTS=weights, MAPLE_VIT=vit and MAPLE_CONNECTOR=connector.safetensors. server.py l
serve.sh runs server.py with MAPLE_WEIGHTS=weights, MAPLE_VIT=vit and MAPLE_CONNECTOR=connector.safetensors. server.py loads MapleLM from the packed weights directory (maple_lm.py _load_packed), QwenViT from vit/vision.safetensors (qwen_vit.py), and the connector state dict from connector.safetensors. This shipped connector file was copied from the trained conn_main.safetensors (trajectory#295-314). Answers, both letter scoring and generation, come from this stack only.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC and OPENAI found nothing. API_KEY matched only the SPIN documentation. Remote compute
Trajectory searches for ANTHROPIC and OPENAI found nothing. API_KEY matched only the SPIN documentation. Remote compute was accessed through spin compute exec and scp with a spin key, which is allowed. I found no step that used, copied or exported harness credentials, and no extra harness CLI was launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 14m into the runtook 5m 40s
startup 77.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 19.0 s after a 0.9 s first question; at that rate the 900 held-out questions take 115 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed3h 1m into the runtook 4m 45s
startup 72.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 72.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 21.9 s after a 0.3 s first question; at that rate the 900 held-out questions take 132 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed3h 36m into the runtook 6m 54s
startup 76.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 76.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 28.8 s after a 0.3 s first question; at that rate the 900 held-out questions take 173 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-20 23:24:39 UTC.
Final submission (/workspace/output/, 110 files, 11.66 GB):
- Language model:
deepgrove/maple-previewternary MoE, frozen and shipped in lossless 2-bit packed form (every checkpoint weight row has exactly magnitudes 0 and α, so it is inherently ternary). - Vision encoder: Qwen2.5-VL-3B vision tower only (frozen, 1.34 GB) — no Qwen language weights shipped and no image-to-text stage.
- Connector: trained 3-layer MLP (34.4 M params) mapping merged patch features into Maple's embedding space, trained 2000 steps on LLaVA-OneVision-Data train splits (162 k rows, MCQ/open/caption mix).
- Serving:
serve.shinstalls vendored wheels offline and starts an OpenAI-compatible server; MC questions are answered by single-token letter scoring, with classifier-free guidance blending the image-conditioned and text-only distributions (weight 0.5) so vision adjusts rather than overrides the language prior; open questions use greedy generation.
Validation evidence (provided samples only): 38.3 % on the 120 MC items through the real served API; the image adds 16 unique correct answers (oracle union 55.8 %), and a held-out-half experiment showed that further fine-tuning on this style memorizes (82 % train) but degrades held-out (27 %), so the original connector was kept. Three self-checks passed — startup 76 s of 300 s, full 150-request batch answered, deadline projection 173 s against the 3600 s allowance.