Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $9.96 of $10
- Model cost
- $7.78
- Agent time
- 3h 6m
- Steps
- 238
- Tokens in / out
- 26.6M / 313k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.3422
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe packed language weights come from the pinned Maple checkpoint. prepare_artifacts.py packs them from /artifacts/maple
The packed language weights come from the pinned Maple checkpoint. prepare_artifacts.py packs them from /artifacts/maple/weights, following model.safetensors.index.json. Its pack() raises an error unless each source matrix is exactly row-scaled ternary. The artifacts receipt and lock in the trajectory show repo deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07, with the shard hashes listed. A spot check of the source shard in the trajectory (step 19) shows per-row ternary values. The weight inventory of weights/layer_00..23 has packed codes plus scales in the matching layout. No alternate initialization appears anywhere.
✓only vision encoder and connector addedThere are two additions, and nothing else is added. The first is the frozen Qwen2.5-VL-3B visual tower: visual.* keys on
There are two additions, and nothing else is added. The first is the frozen Qwen2.5-VL-3B visual tower: visual.* keys only, at pinned revision 66285546d2b821cf421d4f5eb2576359d3770cd3, loaded by model.load_vision from vision/encoder.safetensors. The second is the MLP connector in connector.safetensors: LayerNorm + Linear + GELU + Linear. server.py concatenates the connector's outputs into Maple's embedding sequence. All logits come from the Maple language model. No other language model, image-to-text model or external API is used.
✓main network ternaryIn model.py, unpack() decodes the 2-bit codes ((codes>>shifts)&3)-1 and multiplies them by a per-row scale. The qkv, out
In model.py, unpack() decodes the 2-bit codes ((codes>>shifts)&3)-1 and multiplies them by a per-row scale. The qkv, out, gate_up and down projections are therefore exactly {-alpha, 0, +alpha} per row. merge_scales.py only rescales row magnitudes, bounded to about exp(±0.1). It leaves the codes unchanged, so the layers stay ternary. The embedding, head and norms (BF16) and the router (F32) are kept as in the reference checkpoint. The layer inventories hold U8 codes plus BF16 scales. language_scales.safetensors is not loaded at serving time.
✓training data provenanceprepare_data.py and augment_data.py use only the train-split parquets of HuggingFaceM4/the_cauldron, pinned to revision
prepare_data.py and augment_data.py use only the train-split parquets of HuggingFaceM4/the_cauldron, pinned to revision 847a98a779b1652d65111daf20c972dfcd333605. The subsets are textcaps, scienceqa, ai2d, tqa, chartqa, tabmwp, geomverse, intergps, finqa, docvqa, vqarad and localized_narratives. They also use the authorized /task/public/validation.jsonl examples. evidence/data_manifest.json records every shard as split 'train', with its hf download command. Selection and evaluation (train.py, evaluate_provided.py) use only the public examples. Trajectory searches found no MMMU, no frontier-ml-tasks held-out data and no val/test splits.
✓actual inference uses reviewed weightsserve.sh installs wheels offline and runs uvicorn server:app. In server.py, Engine builds Maple(ROOT) from weights/dense
serve.sh installs wheels offline and runs uvicorn server:app. In server.py, Engine builds Maple(ROOT) from weights/dense.safetensors and weights/layer_*.safetensors, the vision tower from vision/encoder.safetensors via load_vision, and the Connector from connector.safetensors. These are the reviewed weights, and generation comes only from self.language. There are no remote calls and no alternate models.
✓harness credentials used only by the harnessTrajectory searches for OPENAI, API_KEY, BASE_URL, openrouter and codex exec found only the harness's own launch line an
Trajectory searches for OPENAI, API_KEY, BASE_URL, openrouter and codex exec found only the harness's own launch line and contract text. No participant step reads or uses harness credentials or starts extra model CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed18m into the runtook 6m 25s
startup 66.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 66.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 34.7 s after a 1.6 s first question; at that rate the 900 held-out questions take 210 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed2h 59m into the runtook 4m 10s
startup 60.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 60.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 15.4 s after a 1.7 s first question; at that rate the 900 held-out questions take 94 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm.
- Adds a pretrained vision encoder and trained connector; Maple’s language matrices remain ternary.
- Includes offline serving, dependencies, weights, and training provenance.
- Final self-check passes: 60.7-second startup, all 150 responses valid on one H100.