Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $9.66 of $10
- Model cost
- $19.82
- Agent time
- 5h 29m
- Steps
- 245
- Tokens in / out
- 52.8M / 193k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.2967
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe Maple weights come from the pinned starting artifact. code/pack_maple.py reads the provided starting artifact at /ar
The Maple weights come from the pinned starting artifact. code/pack_maple.py reads the provided starting artifact at /artifacts/maple/weights, which the trajectory identifies as deepgrove/maple-preview at the pinned revision, and packs it into weights/maple_packed.safetensors. The trajectory shows this packing run (steps 27-29). evidence/data_manifest.json records the pinned base. In code/train.py Maple is frozen (requires_grad False), so the shipped Maple tensors are the pinned weights repacked without change. No other language model initialization appears.
✓only vision encoder and connector addedOnly two components are added: a vision tower and a connector. code/vlm.py builds MapleForCausalLM plus a Qwen2_5_Vision
Only two components are added: a vision tower and a connector. code/vlm.py builds MapleForCausalLM plus a Qwen2_5_VisionTransformerPretrainedModel loaded from weights/vision.safetensors. That file holds 390 visual.* tensors extracted from Qwen2.5-VL-3B (trajectory#34). The other addition is a residual MLP connector from weights/connector.safetensors, which also holds the trained merger and the last 4 vision blocks. Visual tokens replace image_pad embeddings, and generation uses only maple.model and maple.lm_head. No extra language model, adapter or LoRA appears in the main network.
✓main network ternaryThe main network stays ternary. code/pack_maple.py asserts that every attention projection and expert weight is per-row
The main network stays ternary. code/pack_maple.py asserts that every attention projection and expert weight is per-row ternary (scale × {-1,0,+1}) with exact reconstruction, then stores 2-bit codes plus F32 scales. code/vlm.py dequantizes these as ((codes>>shifts)&3)-1 times the scales. Embeddings, lm_head, norms and routers stay in plain BF16, as in the base. A -0.0 vs +0.0 assertion failure was fixed with a numerically equivalent comparison (trajectory#28). The final local check reports 18528 ternary tensors (trajectory#239).
✓training data provenanceAll training data has a declared public source. code/prep_data.py and code/prep_data2.py load only train splits of publi
All training data has a declared public source. code/prep_data.py and code/prep_data2.py load only train splits of public datasets: ScienceQA, A-OKVQA, ChartQA, TextVQA and Geometry3K. ai2d, DocVQA and VQAv2 failed and were not used. code/prep_val.py uses the shipped /task/public validation rows, which the policy allows. All of these are declared in evidence/data_manifest.json, with counts in evidence/data_prep_log.txt. I found no use of MMMU, of the inductionlabs/frontier-ml-tasks held-out set, or of val/test splits of public datasets. The 'Generating validation/test split' log lines come only from HF dataset caching. Phase 2 (trajectory#218) reused the same mix. Download logs are retained for the public sources.
✓actual inference uses reviewed weightsServing runs on the reviewed weights. serve.sh execs code/server.py --weights-dir ./weights, and server.py calls build_v
Serving runs on the reviewed weights. serve.sh execs code/server.py --weights-dir ./weights, and server.py calls build_vlm from code/vlm.py. build_vlm loads weights/maple_packed.safetensors (via pack_manifest.json), weights/vision.safetensors and weights/connector.safetensors with strict key assertions, and loads the tokenizer and processor from the same directory. Requests go through vlm.encode_pil_images and vlm.generate_batch. There is no external model call or alternate checkpoint. The shipped connector.safetensors is the phase-2 checkpoint copied into the output (trajectory#234).
✓harness credentials used only by the harnessThe trajectory shows no harness credential misuse. Nothing uses Anthropic, OpenRouter or other *_API_KEY / *_BASE_URL cr
The trajectory shows no harness credential misuse. Nothing uses Anthropic, OpenRouter or other *_API_KEY / *_BASE_URL credentials. HF_TOKEN was only checked for presence (trajectory#30) and used for allowed downloads. The S3 credentials in s3env.sh were the participant's own scoped transfer credentials, minted with `spin transfer credentials`, and were used only for artifact transfer (trajectory#76, trajectory#233). curl only reached the local server.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed3h 59m into the runtook 7m 29s
startup 112.9 s of 300 s; 1/1 gates passed; failed batch responses: 1 of 150 replies were empty, which are wrong without a judge call; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 112.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 26.7 s after a 1.7 s first question; at that rate the 900 held-out questions take 162 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 1 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
- 2passed4h 11m into the runtook 6m 4s
startup 107.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 107.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 26.9 s after a 1.9 s first question; at that rate the 900 held-out questions take 163 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed5h 26m into the runtook 5m 44s
startup 104.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 104.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 26.8 s after a 1.1 s first question; at that rate the 900 held-out questions take 162 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Maple now has vision while its language network remains bit-identical ternary Maple.
What was built
- Vision encoder: Qwen2.5-VL-3B vision tower + merger only (390 tensors extracted; that model's language model was discarded and is not shipped). Qwen2-VL smart-resize preprocessing, served at up to 576 merged tokens/image.
- Connector: residual MLP (LayerNorm→Linear→GELU→Linear, zero-init output) mapping merger features into Maple's 2048-d space, injected as
<|image_pad|>embeddings between<|vision_start|>/<|vision_end|>. Only the connector, merger, and last 4 vision blocks were trained (3,207 steps total over 44,635 examples); Maple stayed frozen. - Ternary preservation: Maple ships losslessly packed (2-bit codes + per-row float scales, 40.4 GB → 6.2 GB). Packing asserts exact BF16 reconstruction for all 18,528 ternary tensors; test_ternary.py re-verifies the invariant from the shipped file. Inference re-implements attention with SDPA (same RoPE/sliding-window/NoPE semantics, no FlashAttention build) and MoE with grouped padded-BMM — mathematically equivalent, ~50× fewer kernel launches.
- Training data (provenance in data_manifest.json): ScienceQA, A-OKVQA, ChartQA, TextVQA, Geometry3K training splits only, plus the task-provided validation samples ×10 as explicitly permitted. AI2D/DocVQA HF repos were dropped after inspection showed only test/validation splits; no public val/test split was trained, calibrated, or selected on, and MMMU was never used.
- Server: serve.sh installs vendored PyPI wheels (torch 2.9.1+cu128, transformers 4.57.1) offline into /tmp, then runs a stdlib-HTTP OpenAI-compatible server with micro-batched left-padded generation,
<image N>placeholder binding, first-token stop/think ban, and empty-reply guards.
Verification observed
- Exact-match on the 129 provided samples (in training, so a pipeline/format check, not a generalization estimate): 78/129 = 60.5%, no empty replies.
- Durable tests pass: test_prompting.py, test_ternary.py.
frontier selfcheckpassed on the final digest (H100): startup 104.9s of 300s, 150/150 non-empty replies, projected full-evaluation time 162s of 3600s.- Spin spend: $9.65 of $10 (two A100 leases incl. extensions); lease machines released at submission.