Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $18.43 of $50
- Model cost
- $21.56
- Agent time
- 3h 35m
- Steps
- 342
- Tokens in / out
- 34.9M / 360k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.4044
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe language model was initialized from the task-provided starting checkpoint at /artifacts/maple/weights. The task pins
The language model was initialized from the task-provided starting checkpoint at /artifacts/maple/weights. The task pins that checkpoint in STARTING_ARTIFACT.json as deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 (trajectory#1-13 and trajectory#98 show this). training/convert_maple_ternary.py ternarized it, and both weights/lm/manifest.json and logs/convert_ternary_init.log record the source as /artifacts/maple/weights. The QAT attention latents were extracted directly from /artifacts/maple/weights q/k/v/o (trajectory#98). Training started from these weights (train.py calls build_model with qat_latent_file), and the tokenizer and config were copied from the same artifact.
✓only vision encoder and connector addedTwo components are added. The first is a Qwen2.5-VL-7B visual tower fetched by training/fetch_vit.py: it downloads only
Two components are added. The first is a Qwen2.5-VL-7B visual tower fetched by training/fetch_vit.py: it downloads only the visual.* tensors at a pinned revision, and the merger is stripped when the model is built. The second is an MLP connector. The vit.safetensors inventory contains only blocks/patch-embedding tensors, connector.safetensors contains only connector tensors, and no Qwen language-model tensors or second LM appear anywhere. maple_vlm/vlm.py places the visual embeddings at the image-pad positions inside the Maple LM; there is no image-to-text front end.
✓main network ternaryAttention and expert weights are stored as 2-bit packed U8 codes with a per-row BF16 alpha, in weights/lm/layer_XX.safet
Attention and expert weights are stored as 2-bit packed U8 codes with a per-row BF16 alpha, in weights/lm/layer_XX.safetensors and weights/lm_attn.safetensors. maple_lm.py/loader.py rebuild them with dequant(codes, alpha), so every weight is exactly one of {-alpha, 0, +alpha}. training/export_final.py re-ternarizes the QAT latents before export. The embeddings, lm_head, norms and router stay in higher precision, as in the original Maple, and the manifest zero fractions (~0.4) match a ternary distribution.
✓training data provenanceTraining data was declared in architecture.json, training/prep_data.py and logs/data_manifest.json: pinned revisions of
Training data was declared in architecture.json, training/prep_data.py and logs/data_manifest.json: pinned revisions of the train splits of lmms-lab/LLaVA-OneVision-Data, HuggingFaceM4/the_cauldron and MMInstruction/ArxivQA. The files actually used are train-* parquet files. The only MMMU material used was the 129 shipped /task/public validation rows, used for selection and evaluation, which policy allows. A search of the trajectory found no MMMU download, no frontier-ml-tasks/held-out set and no validation/test splits.
✓actual inference uses reviewed weightsserve.sh runs python -m maple_vlm.server with MAPLE_WEIGHTS=$ROOT/weights. server.py calls build_model, which loads weig
serve.sh runs python -m maple_vlm.server with MAPLE_WEIGHTS=$ROOT/weights. server.py calls build_model, which loads weights/lm (load_packed), lm_attn.safetensors, vit.safetensors (strict) and connector.safetensors, and engine.py runs only that model. The trajectory shows export_final.py producing the step-3597 export, which was copied into output/weights (trajectory#306-307) and matches weights/export_info.json. No alternate model or external API path exists.
✓harness credentials used only by the harnessA search of the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL in participant code; th
A search of the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL in participant code; the one env dump at step 1-13 shows no keys. It also found no claude/codex/gemini CLI sessions and no HF uploads (no push_to_hub). Compute and storage went through Spin CLI and s5cmd, which policy allows. HF_TOKEN was used only for downloads (prep_data.py, fetch_vit.py).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 36m into the runtook 4m 34s
startup 54.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 54.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 27.4 s after a 0.4 s first question; at that rate the 900 held-out questions take 165 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
I've submitted the Maple vision model. The self-check passed on an H100, with startup in 54.7 s against the 300 s limit. All 150 sample replies parsed and were non-empty. The 900-question run projects to about 165 s of the 3,600 s allowed. The architecture review did not run in the self-check, so it is untested. I did not edit /workspace/output between the check's copy and the submission.
The model uses a frozen Qwen2.5-VL vision encoder (the vision tower only) and a trained MLP connector in front of Maple. The language network stays ternary: after training, every attention code is identical to the base model's. Only the per-row scales and norm weights were lightly tuned.
On the 120 multiple-choice rows you provided, the server gets 52 right (43.3%), against about 36.7% for text-only. At 120 rows the standard error is about ±4.5%, so the true accuracy is uncertain. The judged validate.py score on all 129 rows was 0.403. Open numeric questions are answered close to randomly, which I judged not worth optimizing since they are roughly 7% of the data.
Multiple-choice questions are answered by scoring the option letters with the language model, averaged over up to 8 reorderings of the options. Open questions use short greedy generation.
architecture.json lists the training and inference files. It names the three training datasets with revisions and train splits only, in logs/data_manifest.json. It also records that the provided validation rows were used only for monitoring and design choices. No MMMU data was used beyond what you provided.
frontier submit --confirm ended the session and released the Spin machines.