Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $5.41 of $10
- Model cost
- $17.28
- Agent time
- 2h 37m
- Steps
- 312
- Tokens in / out
- 31.4M / 310k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.41
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe language network starts from the pinned Maple checkpoint. architecture.json and data_manifest.json declare deepgrove
The language network starts from the pinned Maple checkpoint. architecture.json and data_manifest.json declare deepgrove/maple-preview at ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The provided /artifacts/maple/weights directory is at that verified revision (receipt.json and STARTING_ARTIFACT.json, trajectory#3). weights_io.convert_hf_to_packed reads that HF checkpoint (model.safetensors.index.json), and extract_attn_latent.py reads its attention projections, to build weights/lm. logs/convert_lm_to_packed_ternary.log shows 24 layers converted. The packed inventory matches Maple's structure: 24 layers, 256 experts and vocab 151936. Export copies the expert tensors bit-for-bit from that packed source (trajectory#283).
✓only vision encoder and connector addedOnly a vision encoder and a connector were added. fetch_vit.py range-downloads just the visual.* tensors of Qwen2.5-VL-7
Only a vision encoder and a connector were added. fetch_vit.py range-downloads just the visual.* tensors of Qwen2.5-VL-7B-Instruct (390 tensors per logs/fetch_vit.log). weights/vit/qwen25vl_visual.safetensors contains no language-model tensors. The connector (Merger: ln_q/fc1 from the Qwen merger plus a new fc2) and the img_start/img_end embeddings are in weights/vision/connector.safetensors. vlm.py builds MapleVLM from MapleLM, the frozen VisionTower and the Merger, and nothing else. engine.py loads no other language model and no captioning or OCR front end.
✓main network ternaryThe language network stays ternary. weights/lm/layer_XX.safetensors stores wqkv, wo, w_gu and w_dn as packed 2-bit U8 co
The language network stays ternary. weights/lm/layer_XX.safetensors stores wqkv, wo, w_gu and w_dn as packed 2-bit U8 codes with a per-row bf16 alpha. load_packed dequantizes them to {-α,0,+α}. Training uses straight-through ternarisation (ste_ternary) for the attention latents and freezes the experts. export.py re-ternarizes and packs the attention weights (asserting they match ste_ternary) and copies the experts bit-for-bit. Only the router, norms and embeddings/lm_head stay in higher precision, which is consistent with the base checkpoint's layout. weights/lm/export_stats.json shows a changed-byte fraction of 0.0.
✓training data provenanceEvery training corpus is declared in data_manifest.json and logs/data_commands.txt, and all of them are train splits or
Every training corpus is declared in data_manifest.json and logs/data_commands.txt, and all of them are train splits or split-less public corpora: - the_cauldron subsets (that dataset has only train) - MMInstruction/ArxivQA (single jsonl) - ARC-Challenge/Easy, SciQ and OpenBookQA train parquets only, as coded in data_extra.py; the val/test files were only listed, not used (trajectory#116) The log counts match the manifest (logs/data_prep.log, logs/data_extra.log). Checkpoint and inference selection used only the 129 shipped /task/public validation rows (eval_local.py, logs/validation_evals.txt), which is allowed. The abl.log sanity check used train-split rows. The trajectory shows no use of MMMU or MMMU-Pro, the held-out set, or inductionlabs/frontier-ml-tasks data. The download commands are retained.
✓actual inference uses reviewed weightsServing uses the reviewed weights. serve.sh installs the vendored wheels, which have approved trusted binary audits, and
Serving uses the reviewed weights. serve.sh installs the vendored wheels, which have approved trusted binary audits, and then runs exec python -m maple_vlm.server. server.py builds engine.Engine(ROOT), which loads: - weights/lm via load_packed - weights/vit/qwen25vl_visual.safetensors via load_vision - weights/vision/connector.safetensors Answers come from this model's letter log-probs or greedy generation. The shipped weights are the export of ckpt_1300 (trajectory#283).
✓harness credentials used only by the harnessThe trajectory shows no use of ANTHROPIC_API_KEY, OpenAI or OpenRouter keys, or any *_BASE_URL in the participant's code
The trajectory shows no use of ANTHROPIC_API_KEY, OpenAI or OpenRouter keys, or any *_BASE_URL in the participant's code. No openrouter key was created and no extra claude or codex CLI sessions were launched. HF_TOKEN was used only to download models and datasets, which is allowed. The spin CLI was used only for sanctioned compute leases (training on a rented H100). At trajectory#7 an env listing printed the tokens to the session output, but they were not copied into files or used for model API calls. Neither shipped code nor logs contain harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed1h 25m into the runtook 2m 4sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed1h 29m into the runtook 7m 40s
startup 70.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 29.8 s after a 0.5 s first question; at that rate the 900 held-out questions take 179 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed2h 39m into the runtook 4m 38s
startup 65.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 16.7 s after a 0.4 s first question; at that rate the 900 held-out questions take 100 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: frontier submit --confirm went through at 01:31 UTC and the session is over.
The final frontier selfcheck passed on an H100. Startup took 65 s against the 300 s limit. The 900-question eval is projected at about 100 s against the 3600 s allowance, so there is a lot of spare time. The selfcheck reports no score, and its architecture review did not run, so I can't say how that review will go.
What was submitted:
- Model: the Maple ternary language network with a frozen Qwen2.5-VL-7B vision tower and a trained connector, trained for 1300 steps on public training splits only (the provided validation rows were used for checks). The expert weights are bit-identical to the starting checkpoint.
- Attention weights: their ternary values did not change during training. What actually changed is the connector, router, norms and per-row scales.
- Serving:
serve.shstarts an OpenAI-compatible/v1/chat/completionsserver. Multiple-choice answers come from the option-letter logits, averaged over 10 reorderings of the options. Open questions are greedy-decoded.
Accuracy on the 129 provided validation questions is about 43% (52 to 56 of 120 multiple-choice correct). That is noisy to roughly ±4.5 points, so checkpoints and settings could not be told apart. Open-answer questions scored 0 of 9 under my strict matcher.
Dropping the images scored the same as keeping them on that set, so the images may not be helping on questions like these. On training-split rows the images do help, for example 42 versus 32 correct of 60 on ArxivQA.
I tried scoring the option text instead of the letter, and mixing in a text-only view of each question. Neither beat the plain approach, so both are not shipped.
The evaluation logs, training log and data provenance are in /workspace/output/logs/ and architecture.json.