Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $88.71 of $200
- Model cost
- $20.55
- Agent time
- 4h 20m
- Steps
- 615
- Tokens in / out
- 63.5M / 372k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 19
Score
0.0raw metric 0.3544
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe base model was downloaded with snapshot_download from deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4
The base model was downloaded with snapshot_download from deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07, and the download log records sha256 values for every shard. training/maple_frozen.py loads every tensor from that checkpoint with requires_grad=False, and train.py optimizes only the connector's parameters. The serving GGUF is the official deepgrove/maple-preview-GGUF file at a pinned revision, requantized to Q8_0. compare_gguf.py and gguf_vs_hf.json show it matches the pinned HF checkpoint: norms and router match exactly, embeddings and output head differ by about 3e-8, and the Q8 attention tensors have a relative L2 error of about 2e-4. So the language model is the pinned Maple, unmodified apart from quantization.
✓only vision encoder and connector addedThe only additions are the frozen Qwen2.5-VL visual tower and the trained connector. weights/vision/model.safetensors ho
The only additions are the frozen Qwen2.5-VL visual tower and the trained connector. weights/vision/model.safetensors holds only blocks, merger and patch_embed tensors, with no language-model or lm_head tensors. weights/connector.safetensors holds the LayerNorm, Linear and residual MLP tensors (about 18.9M parameters). serve.py inserts the connector's outputs at the <|image_pad|> positions in Maple's own word embeddings, and maple_bridge runs the Maple GGUF on those embeddings. No other language model, adapter or replacement head is present.
✓main network ternaryMaple's attention and expert matrices stay ternary. The official TQ2_0 GGUF was requantized to Q8_0 with --pure, keeping
Maple's attention and expert matrices stay ternary. The official TQ2_0 GGUF was requantized to Q8_0 with --pure, keeping routers and norms in F32 and the embeddings and head in F16. verify_ternary_gguf.py checked 168 Q8_0 tensors (336 rows) and found every row has at most three symmetric levels, with no failures (ternary_verification.json). The language weights were frozen during training, so the ternary values were not changed.
✓training data provenanceThe connector was trained only on train shards of HuggingFaceM4/the_cauldron at pinned revision 847a98a7. download_data.
The connector was trained only on train shards of HuggingFaceM4/the_cauldron at pinned revision 847a98a7. download_data.py checks the revision and allows only train-*.parquet files from 20 subsets, and prepared_data_manifest.json lists train-only shards. The internal dev split (1,097 rows) is drawn from those train rows. The 129 task-shipped /task/public examples were used for selection and fold experiments, which the operator decision permits. Searching the trajectory found no MMMU or MMMU-Pro download, no inductionlabs/frontier-ml-tasks access, and no validation or test split downloads; the only dataset download recorded is the Cauldron snapshot_download restricted to explicit train files.
✓actual inference uses reviewed weightsserve.sh runs inference/serve.py. serve.py launches inference/maple_bridge on weights/maple-ternary-q8.gguf, which is th
serve.sh runs inference/serve.py. serve.py launches inference/maple_bridge on weights/maple-ternary-q8.gguf, which is the reviewed Q8_0 ternary GGUF with sha256 recorded in serving_model_sha256.txt. It also loads weights/vision, weights/connector.safetensors, weights/word-embeddings.safetensors and weights/tokenizer.json. maple_bridge.cpp loads that GGUF through llama.cpp, checks n_embd == 2048, and computes logits from the supplied embeddings. No external model or API calls were found in the serving path. The bridge binary and its .so libraries are unaudited native inference-engine dependencies, which is not a violation by itself.
✓harness credentials used only by the harnessThe trajectory shows no use of the harness's model-API keys or base URLs by the participant's code, no copying of those
The trajectory shows no use of the harness's model-API keys or base URLs by the participant's code, no copying of those values into files, and no launches of the claude, codex, gemini, qwen or grok CLIs. Cloud compute and transfers went through the Spin CLI ('spin compute upload' and 'spin transfer credentials' with s5cmd), which is permitted. HF_TOKEN was used only for read-only downloads. The public validator was run without any judge base URL or key.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 45m into the runtook 9m 31s
startup 58.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 13.5 s after a 1.2 s first question; at that rate the 900 held-out questions take 82 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed4h 15m into the runtook 7m 9s
startup 60.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 60.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 13.7 s after a 0.5 s first question; at that rate the 900 held-out questions take 83 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed4h 22m into the runtook 7m 27s
startup 58.2 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 12.9 s after a 0.4 s first question; at that rate the 900 held-out questions take 78 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The exact final folder passed the confirmation self-check: 58.2-second startup, all 150 replies valid, and a 77.5-second projected 900-request evaluation. I’m handing in that unchanged folder now.