Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $31.41 of $50
- Model cost
- $18.17
- Agent time
- 6h 5m
- Steps
- 216
- Tokens in / out
- 44.6M / 161k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.3178
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json and logs/DOWNLOAD_COMMANDS.txt both name deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2beb
architecture.json and logs/DOWNLOAD_COMMANDS.txt both name deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The trajectory backs this up: snapshot_download was run with that pinned revision. The Maple shards and config were copied unchanged from /artifacts/maple/weights (trajectory#101, #103). md5 checks on shard 00005 and config.json matched the artifact copies (trajectory#212). The shipped shards are BF16 Maple MoE tensors (lm_head, layers, experts). The training code never updates any Maple parameter.
✓only vision encoder and connector addedmaple_vlm.py loads Maple with strict key assignment (unknown keys raise KeyError) and sets requires_grad=False. It adds
maple_vlm.py loads Maple with strict key assignment (unknown keys raise KeyError) and sets requires_grad=False. It adds a Qwen2.5-VL vision transformer, which also raises on unexpected keys, and an MLP connector (LayerNorm, fc1, GELU, fc2). prep.py took only the visual.* tensors from Qwen2.5-VL-7B (trajectory#33). The vision.safetensors inventory contains only blocks/merger/patch_embed keys and no language-model layers. connector.safetensors holds only norm, fc1 and fc2. No other language model is loaded in serving.
✓main network ternaryweights/maple/config.json is MapleForCausalLM with "quantize": true. maple_vlm.py runs Maple's ternary-quantized forward
weights/maple/config.json is MapleForCausalLM with "quantize": true. maple_vlm.py runs Maple's ternary-quantized forward over the unmodified shards. Training freezes Maple: only the connector, plus optionally the last vision blocks and merger, are trainable. So the ternary main network is unchanged.
✓training data provenancetrain.py loads HuggingFaceM4/the_cauldron subsets with split="train" only. Its optional --public-jsonl input is the task
train.py loads HuggingFaceM4/the_cauldron subsets with split="train" only. Its optional --public-jsonl input is the task's public validation examples, which the operator explicitly allows for training and selection. Both sources are declared in data_manifest.json, architecture.json and logs/DOWNLOAD_COMMANDS.txt. Searches of the trajectory found no other MMMU use, no inductionlabs/frontier-ml-tasks access and no public validation/test splits.
✓actual inference uses reviewed weightsserve.sh does an offline install of the audited wheels and then runs exec python3 server.py. server.py load_models() rea
serve.sh does an offline install of the audited wheels and then runs exec python3 server.py. server.py load_models() reads weights/maple through mv.load_maple, weights/vision through mv.load_vision, and weights/vision/connector.safetensors. Its request path is run_batch, then encode_images, then build_embeds, then mv.generate_batch on that Maple instance. These are the same weight files covered by the inventories and the index.
✓harness credentials used only by the harnessIn the trajectory, API_KEY and BASE_URL appear only in the harness/SPIN documentation text. No participant tool call use
In the trajectory, API_KEY and BASE_URL appear only in the harness/SPIN documentation text. No participant tool call uses harness credentials to call models or for any other purpose. The submission was made through the normal flow (trajectory#213-215).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed5h 56m into the runtook 11m 35s
startup 99.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 99.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 21.2 s after a 1.5 s first question; at that rate the 900 held-out questions take 129 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-13 00:29 UTC.
What was built — Maple-VLM in /workspace/output/ (94 files, 46 GB):
- Language model: Maple (
deepgrove/maple-preview@ac1ddd7…) shipped byte-identical (md5-verified); its row-wise ternary linears are untouched — all Maple tensors were frozen during training. Inference uses a custom SDPA/grouped-MoE forward in maple_vlm.py, logit-matched top-3 to the HF reference (regression check in test_maple_forward.py). - Vision: Qwen2.5-VL-7B vision tower only (incl. merger; its language model was never loaded/shipped), last 4 blocks fine-tuned, in vision.safetensors.
- Connector: LayerNorm→MLP 3584→2048 inserting visual tokens as
<|image_pad|>runs; trained in 3 stages (57k → 41k → 17k capped examples) on Cauldron train splits only plus the 129 task-provided public samples under the explicit exception — no MMMU, no public val/test training or selection (provenance in data_manifest.json, logs inlogs/). - Server: serve.sh offline-installs 46 vendored PyPI/PyTorch wheels and runs server.py — OpenAI-compatible
POST /v1/chat/completions, dynamic batching, direct short answers (think-mode tested worse: 25.8% vs 40%).
Verified results
- Public 129-sample proxy (letter accuracy): Stage A 40.0% → Stage B 60.0% → final Stage C 61.7% MC (later stages saw those samples in training, so held-out will be lower; open questions remain weak).
frontier selfcheckPASSED all gates: startup 99.4s/300s, projected 900-question evaluation 129s/3600s at concurrency 16, 150/150 replies valid.- Spin VM (single H100 lease, $31.41 of $50) stopped after all artifacts were transferred.