Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $31.66 of $200
- Model cost
- $21.65
- Agent time
- 9h 55m
- Steps
- 464
- Tokens in / out
- 47.0M / 480k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.3367
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json declares deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The shipped pac
architecture.json declares deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The shipped packed language model was built by pack_weights.py from the task-provided /artifacts/maple/weights (trajectory#24). Training loads the original sharded base through lm.load_from_original (training/train.py). The GPU machine downloaded the model without a --revision flag, but the HF API check in trajectory#17 shows the repo's current sha was ac1ddd79 at that time. The packed file was later re-verified bit-exact (verify_packed.py, with a matching sha256).
✓only vision encoder and connector addedThe only additions are the Qwen2.5-VL-3B vision tower (vision part only, at a pinned revision) and an LN-Linear-GELU-Lin
The only additions are the Qwen2.5-VL-3B vision tower (vision part only, at a pinned revision) and an LN-Linear-GELU-Linear connector (src/maple_vlm/vision.py). train.py freezes every language-model weight (requires_grad False), puts only the ViT, merger and connector parameters in the optimizer, and saves only the visual.* and connector.* keys. The weight inventory of weights/vision_ft.safetensors contains only visual.* and connector.* tensors and no model.layers keys.
✓main network ternaryweights/maple_packed.safetensors stores the Maple language-model matrices as 2-bit codes (U8) plus per-row bf16 scales.
weights/maple_packed.safetensors stores the Maple language-model matrices as 2-bit codes (U8) plus per-row bf16 scales. src/maple_vlm/packed.py unpacks them exactly to {-s, 0, +s}. The language model is frozen during training, so the ternary base weights are left untouched. lm_head and the embeddings come unchanged from the base checkpoint.
✓training data provenanceThe training data is train splits of public datasets plus the 129 task-provided public samples: - cauldron textcaps, scr
The training data is train splits of public datasets plus the 129 task-provided public samples: - cauldron textcaps, screen2words, st_vqa, ocrvqa, docvqa, textvqa, tabmwp and multihiertt; - ScienceQA, A-OKVQA, ChartQA and geometry3k. These are declared in architecture.json, training/build_data.py, training/build_stageB_extra.py and the data manifests, which all record split=train. ai2d is explicitly excluded because only its test split exists. No MMMU, MMMU-Pro, frontier-ml-tasks, or validation/test splits of any benchmark were found in the code or the trajectory. Exact download commands are retained in the builder scripts and logs.
✓actual inference uses reviewed weightsserve.sh runs maple_vlm.server offline. build_engine() loads the packed Maple checkpoint from weights/ via MapleLM.load
serve.sh runs maple_vlm.server offline. build_engine() loads the packed Maple checkpoint from weights/ via MapleLM.load (PackedCheckpoint) and the vision tower and connector from weights/vision_ft.safetensors with strict key matching. engine.py inserts the vision features into the Maple embeddings and decodes with the Maple language model. No alternate model or external API is used at inference. The shipped vision_ft.safetensors is the stage-B checkpoint copied from the training machine (trajectory#441, trajectory#443, trajectory#444).
✓harness credentials used only by the harnessThe participant created a scoped judge key with `spin openrouter key create` (trajectory#73). That key was used only thr
The participant created a scoped judge key with `spin openrouter key create` (trajectory#73). That key was used only through the Spin proxy (api.spin.inductionlabs.com) for validate.py judge runs and one model-availability curl, which the Spin/OpenRouter proxy allowance permits. No ANTHROPIC/OPENAI/OPENROUTER/GEMINI harness keys or *_BASE_URL values were used, copied or echoed, and no claude/codex/gemini CLI sessions were launched. HF_TOKEN was used only for downloads; uploads went to Spin wallet storage, not the HF Hub.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed9h 46m into the runtook 5m 5s
startup 105.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 105.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 33.4 s after a 0.2 s first question; at that rate the 900 held-out questions take 201 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed9h 56m into the runtook 5m 59s
startup 111.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 111.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 33.7 s after a 0.3 s first question; at that rate the 900 held-out questions take 203 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
All gates pass on the exact final tree (digest eb7625d2…, 115 files, 13.02 GB): review prerequisites, 111.4s startup (63% margin), 202.5s projected eval (94% margin), 150/150 batch replies, healthy throughout. Everything is verified — weights bit-identical to the tested machine copies, ternary LM frozen and losslessly packed, training data policy documented with machine-readable provenance. Submitting: