Vision for a ternary LM
- Score
- 6.5
- Wallet spend
- $26.39 of $50
- Model cost
- $54.88
- Agent time
- 5h 17m
- Steps
- 202
- Tokens in / out
- 53.0M / 237k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 1st of 14
Score
6.5raw metric 0.4744
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe session downloaded deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 into /dat
The session downloaded deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 into /data/maple (trajectory#55, completion logged in trajectory#56). tools/pack_lm.py turned that checkpoint into weights/lm/maple_ternary.safetensors, and train/COMMANDS.md and trajectory#194-202 record that step. Training loads the same checkpoint through MapleLM.from_hf. The packed inventory has Maple's shapes: 24 layers, 256 stacked experts, 2048 hidden, a 151936-row lm_head. train/logs/verify_backbone.log reports that all 18651 tensors match backbone_contract.json, with 0 mismatches against a fresh ternarization of the base weights. The tokenizer and config files were fetched at the same revision (trajectory#91).
✓only vision encoder and connector addedThe only new parts are the Qwen3-VL-30B-A3B ViT and its mergers, in weights/vision/vision.safetensors (blocks, patch_emb
The only new parts are the Qwen3-VL-30B-A3B ViT and its mergers, in weights/vision/vision.safetensors (blocks, patch_embed, pos_embed, merger and deepstack mergers; no language-model layers). The connector (proj, ds_proj×3, img_start/img_end, plus fine-tuned mergers) is in weights/vision/connector.safetensors. In maple_vl/engine.py, the visual embeddings are written into Maple's input embedding positions and fed as deepstack residuals, then decoded by Maple's own forward pass and lm_head. There is no second language model and no image-to-text front end. The Qwen embedding table is used only offline to fit the least-squares initialization and is not shipped.
✓main network ternarymaple_vl/lm.py from_packed loads 2-bit packed .tq codes with per-row bf16 .alpha for every attention q/k/v/o projection
maple_vl/lm.py from_packed loads 2-bit packed .tq codes with per-row bf16 .alpha for every attention q/k/v/o projection and every expert projection. maple_vl/ternary.py dequantizes them to {-1,0,+1}·alpha. Only the router, norms, embeddings and lm_head are bf16, as in the base model. The language network stays frozen during connector training (train/train_connector.py). train/logs/verify_backbone.log reports 0 non-ternary rows.
✓training data provenanceTraining data is lmms-lab/LLaVA-OneVision-Data at pinned revision 7ca5e5bf…, train split only. Every file in train/logs/
Training data is lmms-lab/LLaVA-OneVision-Data at pinned revision 7ca5e5bf…, train split only. Every file in train/logs/data_manifest.json and data_manifest_phase2.json is a train-*.parquet shard. train/prep_data.py downloads with hf_hub_download and has no val/test split arguments and no MMMU loader. Inference-setting selection used only the 129 provided /task/public validation samples (train/logs/inference_selection.md, tools/eval_val.py), which the task allows. Searches found no MMMU or MMMU-Pro use, no frontier-ml-tasks access, and no other benchmark val/test split anywhere in the shipped files or the trajectory.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline, then execs maple_vl.server --root weights. In maple_vl/server.py and mapl
serve.sh installs the vendored wheels offline, then execs maple_vl.server --root weights. In maple_vl/server.py and maple_vl/engine.py, the Engine loads MapleLM.from_packed(weights/lm), VisionEncoder from weights/vision/vision.safetensors and the Connector from weights/vision/connector.safetensors, and runs prefill and decoding on these weights only. The server makes no external API calls. The shipped connector is the phase-2 final checkpoint copied in trajectory#195 (sha256 12d8ef13…). Those same files are the ones in the reviewed inventories.
✓harness credentials used only by the harnessThe trajectory and shipped files contain no ANTHROPIC, OPENAI, OPENROUTER or GEMINI keys or *_BASE_URL. Local scoring ra
The trajectory and shipped files contain no ANTHROPIC, OPENAI, OPENROUTER or GEMINI keys or *_BASE_URL. Local scoring ran without --judge-base-url, and no claude, codex or gemini CLI session was launched. The only credentials used were HF_TOKEN for Hugging Face downloads and Spin's compute/S3 transfer tooling (trajectory#55, trajectory#90-94), both of which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 32m into the runtook 5m 46s
startup 29.5 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 29.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 109.2 s after a 2.5 s first question; at that rate the 900 held-out questions take 658 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed5h 23m into the runtook 7m 47s
startup 25.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 25.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 270.7 s after a 3.0 s first question; at that rate the 900 held-out questions take 1,627 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The submission is complete and verified. Running the two-step submit.