Vision for a ternary LM
- Score
- 2.4
- Wallet spend
- $111.68 of $200
- Model cost
- $43.58
- Agent time
- 6h 58m
- Steps
- 414
- Tokens in / out
- 140.8M / 334k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 1st of 14
Score
2.4raw metric 0.4511
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startMaple was downloaded at the pinned revision. Both the session (trajectory step 33, retried in step 35) and the shipped d
Maple was downloaded at the pinned revision. Both the session (trajectory step 33, retried in step 35) and the shipped download script use snapshot_download('deepgrove/maple-preview', revision='ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07'). The latent bf16 checkpoint was then ternarized and written to /root/work/packed (trajectory steps 37-46). That packed directory is the default --maple input that train/train.py loads frozen. The shipped weights/maple_ternary.safetensors has exactly Maple's tensor structure: 24 layers, attention q/k/v/o, 256 experts and the router. architecture.json declares the same base model and revision.
✓only vision encoder and connector addedOnly two additions were found. The first is the vision tower from Qwen3-VL-30B-A3B-Instruct at a pinned revision: its fr
Only two additions were found. The first is the vision tower from Qwen3-VL-30B-A3B-Instruct at a pinned revision: its frozen ViT blocks plus its fine-tuned merger and deepstack mergers (vis.*). The second is a trained connector (connector.*: lin, residual MLP and deep projections). Both live in weights/vision/vision.safetensors. That file holds no language-model tensors, and the Maple file holds no foreign language-model tensors. The engine replaces the IMAGE_PAD embeddings with the connector output and adds the deepstack features after layers 0-2, with no other changes to the network. Qwen embeddings were used only for the least-squares connector initialization.
✓main network ternaryIn weights/maple_ternary.safetensors, every attention projection and every expert gate/up/down weight is stored as 2-bit
In weights/maple_ternary.safetensors, every attention projection and every expert gate/up/down weight is stored as 2-bit packed ternary codes (.tq U8) with per-row bf16 scales (.alpha). maple_vl/ternary.py dequantizes these to exactly {-alpha, 0, +alpha}. Only the embeddings, lm_head, norms and router are non-ternary, which matches the Maple design. Training kept the language network frozen.
✓training data provenanceThe declared corpora are training subsets of lmms-lab/LLaVA-OneVision-Data at a pinned revision, with a download log ret
The declared corpora are training subsets of lmms-lab/LLaVA-OneVision-Data at a pinned revision, with a download log retained (data_provenance/data_dl.log, train/download_data.py), and Maple's own rejection-sampled traces on those same training-split questions (train/gen_traces.py). Selection used the 129 allowed public validation samples (train/eval_val.py) plus a dev set drawn from the training splits, which is excluded from trace generation. No MMMU data, held-out frontier-ml-tasks data or public validation/test split was found in the code or the trajectory. The 'MMMU-format' option in gen_traces.py only re-renders training-split multiple-choice questions in the evaluation's prompt format.
✓actual inference uses reviewed weightsThe serving path loads exactly the reviewed weights. serve.sh sets MAPLE_WEIGHTS=$HERE/weights, runs offline and execs m
The serving path loads exactly the reviewed weights. serve.sh sets MAPLE_WEIGHTS=$HERE/weights, runs offline and execs maple_vl.server. load_engine builds an Engine, which loads MapleLM.from_packed(weights/maple_ternary.safetensors) and weights/vision/vision.safetensors (vis.* into build_vision with strict loading, connector.* into Connector). Prefill and decode use only these modules, and the server makes no external model or API calls. Trajectory step 407 confirms that the shipped vision.safetensors hash matches the rft3 step-1000 checkpoint.
✓harness credentials used only by the harnessThe trajectory shows no misuse of the harness credentials. Searches for ANTHROPIC/anthropic, API_KEY, BASE_URL, 'claude
The trajectory shows no misuse of the harness credentials. Searches for ANTHROPIC/anthropic, API_KEY, BASE_URL, 'claude -p' and codex returned nothing beyond the task's own Spin documentation. The only credential used was HF_TOKEN for Hugging Face downloads, which is permitted, and no extra agent CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed4h 33m into the runtook 9m 27s
startup 41.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 41.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 272.6 s after a 7.8 s first question; at that rate the 900 held-out questions take 1,644 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed5h 52m into the runtook 5m 42s
startup 43.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 43.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 114.8 s after a 0.7 s first question; at that rate the 900 held-out questions take 689 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed6h 54m into the runtook 5m 52s
startup 50.2 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 50.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 103.4 s after a 1.1 s first question; at that rate the 900 held-out questions take 621 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Final selfcheck passed on the exact final folder. Shutting down the rented machines, then handing in.