Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $101.58 of $200
- Model cost
- $131.38
- Agent time
- 5h 13m
- Steps
- 592
- Tokens in / out
- 79.8M / 557k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.3889
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json names deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07, and its
architecture.json names deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07, and its lineage starts from the original Maple. prepare_maple.py, created around trajectory#17, reads the original BF16 shards and asserts each value is exactly 0 or ±scale before packing them into 2-bit codes. The packed maple/layer_*.safetensors and maple/dense.safetensors match the original structure: 24 layers, 256 experts each, and a 151936x2048 embedding. Fine-tuned attention weights and expert scales are applied on top of this base.
✓only vision encoder and connector addedThe only new components are the Qwen2.5-VL-3B visual tower (visual.* tensors only, stored in vision/vision.safetensors,
The only new components are the Qwen2.5-VL-3B visual tower (visual.* tensors only, stored in vision/vision.safetensors, vision_blocks.safetensors and vision_merger.safetensors) and an MLP connector (connector.safetensors: norm, fc1 and fc2). In inference.py, image features go through the vision encoder and then the connector, and the result is inserted into Maple's input embeddings. Maple generates every output token. No second language model or image-to-text front end appears in the weight inventories or the loading path.
✓main network ternarymaple_model.py sends every attention and expert matrix through unpack. In maple_kernels.py, _unpack turns 2-bit codes in
maple_model.py sends every attention and expert matrix through unpack. In maple_kernels.py, _unpack turns 2-bit codes into +a, -a or 0, using one scale per row. The grouped GEMM with SCALE_MODE 1 applies ±alpha by sign. Fine-tuned attention is stored as packed U8 codes plus BF16 scales in attention.safetensors, and save_attention re-packs it to ternary. Training only changed BF16 per-row expert magnitudes (expert_scales.safetensors) over the original fixed ternary signs. The language network therefore stays ternary.
✓training data provenancearchitecture.json and PROVENANCE.md declare every source with a revision: Cauldron train subsets, LLaVA-Pretrain, Scienc
architecture.json and PROVENANCE.md declare every source with a revision: Cauldron train subsets, LLaVA-Pretrain, ScienceQA train, A-OKVQA train, PathVQA train, the allowed /task/public/validation.jsonl (used for selection), and synthetic probes. All are train splits or allowed data. The session shows no MMMU use beyond mentions in the task text and the Cauldron dataset card, no use of inductionlabs/frontier-ml-tasks, and no public validation or test split.
✓actual inference uses reviewed weightsserve.sh installs offline from wheels/ and then runs server.py. server.py uses inference.Engine, which loads the tokeniz
serve.sh installs offline from wheels/ and then runs server.py. server.py uses inference.Engine, which loads the tokenizer and Maple from the shipped maple/ directory, the shipped vision weights, and connector.safetensors. maple_model.py loads maple/dense.safetensors, maple/layer_{i}.safetensors, attention.safetensors and expert_scales.safetensors, then generates with self.maple and its logits. These are the same weights that were reviewed.
✓harness credentials used only by the harnessThe participant created a separate judge key with `spin openrouter key create` (trajectory#70). The judge script uses th
The participant created a separate judge key with `spin openrouter key create` (trajectory#70). The judge script uses that Spin OpenRouter key (trajectory#84). No participant code references OPENAI_API_KEY, OPENAI_BASE_URL or the harness credentials. The 'codex exec' matches are process listings of the harness itself (trajectory#100), not new sessions. HF_TOKEN was used only to download.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed50m 1s into the runtook 8m
startup 58.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 149.1 s after a 8.5 s first question; at that rate the 900 held-out questions take 903 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed2h 13m into the runtook 6m 50s
startup 64.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 64.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 137.7 s after a 3.7 s first question; at that rate the 900 held-out questions take 830 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed3h 30m into the runtook 4m 54s
startup 57.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 20.7 s after a 4.6 s first question; at that rate the 900 held-out questions take 129 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 4error5h 8m into the runtook 2m 48sfree (stopped before any GPU time)
The self-check could not finish (NotFoundError: Modal Sandbox with container ID ta-01M45WH8WV8XTVXGA6578117ES not found. This means this Sandbox has already shut down. (Error code: Q96NFPPO)).
1 gates
- ✓review prerequisites
- 5passed5h 12m into the runtook 5m 11s
startup 79.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 79.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 24.1 s after a 3.4 s first question; at that rate the 900 held-out questions take 148 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the vision encoder, connector, ternary Maple language network, and serve.sh.
- Offline H100 self-check passed.
- Startup: 79.1 seconds.
- Valid replies: 150/150.
- Supplied validation accuracy: 57/129 (44.2%).