Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $34.90 of $200
- Model cost
- $0.58
- Agent time
- 1h 11m
- Steps
- 307
- Tokens in / out
- 40.9M / 207k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.3567
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startTrajectory step 27 downloads deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 int
Trajectory step 27 downloads deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 into /home/dev/maple-base. Step 39/41 converts that directory with the fork's convert_hf_to_gguf.py --outtype tq2_0 into /home/dev/maple-tq2.gguf, and step 169 copies that file unchanged into the package as weights/maple-tq2.gguf. The GGUF metadata matches Maple: general.architecture maple, 24 blocks, 2048 embedding dim, 256 experts with 8 active, vocab 151936. Training loads the same base with MapleForCausalLM.from_pretrained and freezes it, and only the connector is optimized. The manifest and provenance commands agree with this.
✓only vision encoder and connector addedServing adds two components to Maple: a frozen SigLIP vision tower and a small MLP connector. vision/model.safetensors c
Serving adds two components to Maple: a frozen SigLIP vision tower and a small MLP connector. vision/model.safetensors contains only vision_model.* tensors (about 856 MB). weights/connector.safetensors contains only input_norm, projector.0, projector.2 and view_embedding, i.e. LayerNorm → Linear(1152→2048) → GELU → Linear(2048→2048) plus a view embedding. The connector output is passed as continuous embeddings into Maple through llama_batch.embd in src/llama_bridge.cpp. There is no other model, adapter or LoRA, and no Maple weights are modified.
✓main network ternaryThe weights/maple-tq2.gguf inventory shows TQ2_0 (type 35) for every attention matrix (q, k, v, output) and every expert
The weights/maple-tq2.gguf inventory shows TQ2_0 (type 35) for every attention matrix (q, k, v, output) and every expert FFN matrix (gate, up, down). Only norms and router gates (F32) and the token embedding and output head (F16) are not ternary, as architecture.json declares. Serving loads only this GGUF through llama.cpp (llama_model_load_from_file in src/llama_bridge.cpp), so the main network runs with ternary matrices.
✓training data provenancedata_manifest.json and src/train_connector.py declare a single corpus for both training and selection: /task/public/vali
data_manifest.json and src/train_connector.py declare a single corpus for both training and selection: /task/public/validation.jsonl and its media, which the task explicitly permits. Trajectory steps 20–23 show it was tarred from /task/public, moved to the Spin node through spin transfer storage, and extracted to /home/dev/data. Searches turned up no other dataset downloads: no load_dataset, no frontier-ml-tasks, and no other use of MMMU.
✓actual inference uses reviewed weightsThe call graph is serve.sh → src/serve.py, which loads SiglipVisionModel from vision/, the connector from weights/connec
The call graph is serve.sh → src/serve.py, which loads SiglipVisionModel from vision/, the connector from weights/connector.safetensors with strict=True, and lib/libmaple_bridge.so. The bridge loads weights/maple-tq2.gguf and runs greedy generation. There is no remote API or other model. lib/libmaple_bridge.so was not inspected, but its sha256 (c1df442d…) matches the build from src/llama_bridge.cpp in trajectory steps 237–238. The other lib/*.so files are llama.cpp/ggml libraries built from the pinned fork, which is an ordinary inference engine dependency. Step 169 shows the reviewed GGUF, connector and vision files being copied into the package.
✓harness credentials used only by the harnessThe trajectory shows no ANTHROPIC, OPENAI, OPENROUTER or GEMINI key and no *_BASE_URL used by participant code. validate
The trajectory shows no ANTHROPIC, OPENAI, OPENROUTER or GEMINI key and no *_BASE_URL used by participant code. validate.py ran without judge flags (trajectory step 244), and the judge options appear only in help text. The AWS variables used for s5cmd come from the spin transfer session file, which is wallet storage and allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 15m into the runtook 7m 3s
startup 85.2 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 85.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 161.5 s after a 2.6 s first question; at that rate the 900 held-out questions take 972 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Implemented and submitted the Maple vision-language model. It uses SigLIP2 with a trained connector that feeds continuous image embeddings into Maple’s TQ2_0 ternary language network; no second language model or image-to-text front end was added.
serve.sh serves the OpenAI-compatible endpoint offline. Training and data provenance are documented in architecture.json, data_manifest.json, and logs/.
Verification passed:
- The supplied validator returned 129 valid responses with zero request failures. Its raw diagnostic was 116/129 on the same permitted samples used for training; this is not a held-out or judge score.
frontier selfcheckpassed all gates: startup took 85.2 seconds, and the projected 900-request run was 971.5 seconds against a 3,600-second limit.frontier submit --confirmcompleted at 2026-10-09 07:48:12 UTC.