Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $5.94 of $10
- Model cost
- $1.25
- Agent time
- 4h 11m
- Steps
- 453
- Tokens in / out
- 44.4M / 391k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 19
Score
0.0raw metric 0.3289
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned maple startSTARTING_ARTIFACT.json pins deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 in /artifacts/m
STARTING_ARTIFACT.json pins deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 in /artifacts/maple/weights (read at trajectory#3). Trajectory step 129-148 runs `convert_maple.py --src /artifacts/maple/weights --out .../maple_packed`, so the starting point is that pinned checkpoint. evidence/convert_maple.py only repacks the HF tensors (word_embeddings, q/k/v/o_proj, experts, gate) into 2-bit codes plus alphas. The language model stays frozen during training (requires_grad False in train.py and maple_model.py). The packed layer inventories match the Maple layout: 24 layers, 256 experts, hidden size 2048.
✓only vision encoder and connector addedOnly two components are added. vision/visual.safetensors holds just the Qwen3-VL vision tower (patch_embed, pos_embed, b
Only two components are added. vision/visual.safetensors holds just the Qwen3-VL vision tower (patch_embed, pos_embed, blocks.*, merger.*, deepstack_merger_list.*), with no lm_head or embed_tokens. connector.pt holds only connector.* MLP tensors. server.py runs vision → connector → index_copy into the image-pad slots → the Maple LM, and generates with that LM alone. No second language model or image-to-text front end is involved. The participant looked at the full Qwen3-VL checkpoint during the session but shipped only the visual tensors.
✓main network ternaryThe maple_packed/layer_XX.safetensors files store attention and expert weights as U8 2-bit codes with one BF16 alpha per
The maple_packed/layer_XX.safetensors files store attention and expert weights as U8 2-bit codes with one BF16 alpha per row. maple_model.py's pack_ternary asserts the weights are exactly ternary, and unpack_ternary rebuilds ((code)-1)*alpha, i.e. {-a, 0, +a}. The LM parameters are frozen in both train.py and server.py, so no ternary weight is modified. Embeddings, norms and the router stay bf16 as in the original Maple model.
✓training data provenanceThe training corpus is declared in architecture.json and DATA_MANIFEST.md: 19 LLaVA-OneVision-Data configs, train split
The training corpus is declared in architecture.json and DATA_MANIFEST.md: 19 LLaVA-OneVision-Data configs, train split only, 154,992 samples. evidence/dataset_download.py and its log fetch `<config>/train/0000.parquet` from the HF parquet mirror, and evidence/prep_data.py builds the shards. These were downloaded with HF_TOKEN; the log shows the files but the mirror contents themselves are unverified here. Checkpoint selection used only the 129 supplied public samples (evidence/evalval.py reads public/validation.jsonl). train_p2.py is the shipped train.py with an added keep-open sampling flag, trained on the same shards (trajectory#356). The trajectory mentions MMMU or the held-out set only when quoting the rules; nothing from MMMU or the held-out set was downloaded, and no validation or test split was used.
✓actual inference uses reviewed weightsserve.sh installs the bundled wheels and execs server.py. server.py loads tok/tokenizer.json, the maple_packed/{global,l
serve.sh installs the bundled wheels and execs server.py. server.py loads tok/tokenizer.json, the maple_packed/{global,layer_XX}.safetensors files through unpack_ternary, VisionTower from vision/, and the Connector from connector.pt. Answers come from this pipeline only, through letter() for multiple-choice or generate() for open questions. These are the same weight files covered by the inventories above. The trajectory shows connector.pt being copied into the output directory and the shipped server being run on it for the final check.
✓harness credentials used only by the harnessTrajectory searches found no participant use of ANTHROPIC/OPENAI API keys or harness base URLs, no `claude` or codex ses
Trajectory searches found no participant use of ANTHROPIC/OPENAI API keys or harness base URLs, no `claude` or codex sessions, and no judge-base-url runs. The only credentials used were spin CLI compute and transfer credentials and HF_TOKEN for downloads, which the policy allows. The shipped server makes no external model API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 28m into the runtook 5m 37s
startup 74.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 37.7 s after a 1.6 s first question; at that rate the 900 held-out questions take 228 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed4h 20m into the runtook 4m 27s
startup 65.2 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 30.3 s after a 0.8 s first question; at that rate the 900 held-out questions take 183 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state:
What shipped — deepgrove/maple-preview (packed 2-bit ternary language model, unpacked to bf16 at load) + frozen Qwen3-VL-8B ViT (reimplemented in plain PyTorch, numerically verified) + a 25.2M-parameter connector trained on answer tokens only. The language network is untouched ternary; no second LM, no image-to-text front end.
Training — 50,032 samples from 20 configs of the public lmms-lab/LLaVA-OneVision-Data train split (parquet branch), three phases on one A100. Phase 1 (lr 1e-3, 1400 steps) was the best; the later warm restarts lowered training loss but not validation accuracy, so the phase-1 checkpoint was shipped.
Selection (only the 129 supplied validation samples): 44.2% multiple-choice (53/120), 41.1% overall. Rejected after measurement: training-prior letter correction (41.7%), pixel-scale averaging (no gain), weight soup (41.7%), and a 2-checkpoint probability ensemble (45.8%, but a 2-question margin at double inference cost). The 9 open questions were answered wrong by every checkpoint (0/9) — reported as such.
Verification — real HTTP through server.py with the shipped bytes: 129/129 requests HTTP 200, 0 failures, 0.31 s/question, replies byte-identical to the direct-engine run. frontier selfcheck passed all gates on an H100: startup 65.2 s of 300 s, first POST 0.805 s, projected 900-request evaluation 182.6 s of 3600 s, all 150 replies parsed.
One caveat worth stating plainly: with 120 multiple-choice samples the ±4.5% sampling noise means the 44.2% vs 40.8%/40.0% ranking of the three checkpoints is not statistically strong — the choice followed the task's stated selection criterion rather than a proven difference in ability.