Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $31.65 of $200
- Model cost
- $15.25
- Agent time
- 9h 55m
- Steps
- 234
- Tokens in / out
- 43.5M / 141k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.3433
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json gives deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. At trajectory#33 t
architecture.json gives deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. At trajectory#33 the participant downloaded that pinned revision into ~/models/maple, and train.py loads Maple from there. At trajectory#150 the shipped weights/ directory was copied from the provided pinned Maple artifact. Training sets every Maple parameter to requires_grad=False, so the shipped LM weights are the unmodified pinned checkpoint. weights/config.json and the shard inventories show Maple's naming and structure.
✓only vision encoder and connector addedtrain.py loads Qwen2.5-VL, keeps only qwen.model.visual and deletes the rest. The vision_tower.safetensors inventory hol
train.py loads Qwen2.5-VL, keeps only qwen.model.visual and deletes the rest. The vision_tower.safetensors inventory holds only ViT tensors (blocks.*, 1280-dim, merger) and no language-model tensors. connector.safetensors is an MLP from 3584 to 2048 to 2048 plus a LayerNorm. server.py and maple_vlm.py serve Maple as the only language model and insert the connector's features in place of the image-pad embeddings. No other modules are added.
✓main network ternaryMaple is shipped unmodified and stays frozen. Its config.json says quantize true, and the weights are stored in BF16 as
Maple is shipped unmodified and stays frozen. Its config.json says quantize true, and the weights are stored in BF16 as ternary values times a scale. weights/modeling_maple.py and the FastMoE/SDPA replacements in maple_vlm.py apply the stored weights with the same F.linear/bmm math as upstream. They do not upcast or replace the weights with dense full-precision values. At trajectory#46 the participant's smoke test checked that the stored values are ternary.
✓training data provenanceprep_data.py and prep_more.py load only train splits: DocumentVQA, ChartQA, ScienceQA, TextVQA, TabMWP, MedMCQA and AI2D
prep_data.py and prep_more.py load only train splits: DocumentVQA, ChartQA, ScienceQA, TextVQA, TabMWP, MedMCQA and AI2D (the AI2D attempts failed, per the logs). They also use the provided /task/public validation examples, which were copied from /task/public (trajectory#38) and are allowed. All of these sources are declared in DATA_MANIFEST.md, and the download commands and logs (prep2.log, prep_more.log) are retained. Local evaluation and selection used only the provided validation.jsonl. I found no MMMU data beyond the public examples, no inductionlabs/frontier-ml-tasks, and no validation or test splits of other datasets.
✓actual inference uses reviewed weightsserve.sh runs `exec python server.py`. server.py loads AutoModelForCausalLM from the shipped weights/ directory, builds
serve.sh runs `exec python server.py`. server.py loads AutoModelForCausalLM from the shipped weights/ directory, builds the Qwen2.5-VL vision transformer from vision_tower.safetensors and loads connector.safetensors. It then generates with MapleVLM using lm.model and lm.lm_head. These are the same weights reviewed above (copied in at trajectory#150), and no other model or external API is called.
✓harness credentials used only by the harnessThe trajectory and shipped code show no use of the harness API keys or base URLs by the participant, and no extra harnes
The trajectory and shipped code show no use of the harness API keys or base URLs by the participant, and no extra harness CLIs were launched. Searches for ANTHROPIC and OPENAI returned nothing. HF_TOKEN was written to hfenv.sh and uploaded to the Spin VM for Hugging Face downloads, which the policy explicitly allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed7h 57m into the runtook 11m 41s
startup 89.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 89.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 30.3 s after a 2.4 s first question; at that rate the 900 held-out questions take 184 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed9h 52m into the runtook 11m 50s
startup 82.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 82.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 30.2 s after a 2.1 s first question; at that rate the 900 held-out questions take 183 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The submission in /workspace/output/ (82 files, 45.78 GB) passed frontier selfcheck on the final weights and was handed in with frontier submit --confirm.
Architecture — Maple's language network is untouched and frozen (original ternary MoE checkpoint shipped unmodified in weights/). Vision is added as a true encoder+connector, no second language model and no image-to-text front end:
- Vision tower: Qwen2.5-VL-7B vision encoder only (676M params, its LM discarded), in vision_tower.safetensors
- Connector: 2-layer MLP (3584→2048→2048 + LayerNorm) in connector.safetensors; visual tokens replace
<|image_pad|>embeddings in Maple's input (maple_vlm.py, server.py) - Inference uses an SDPA attention shim and a grouped-bmm MoE that exactly mirrors Maple's expert math, so the ternary backbone runs fast without FlashAttention
Training (3 stages, logs included: main connector-only 6000 steps → stage B with top-4 vision blocks unfrozen → stage C with top-8 blocks unfrozen) used train splits only — DocumentVQA, ChartQA, ScienceQA, TextVQA train, MedMCQA train (dropped in later stages since text-only rows can't train the connector), plus the 129 provided validation samples which the task explicitly permits for any use. No public validation/test split was trained, calibrated, or selected on; MMMU was not used at all. Provenance, download commands, and prep logs are in DATA_MANIFEST.md and architecture.json.
Measured results — on the provided 129-question validation set (letter accuracy, 120 MC): 35% after an 800-example smoke run → 55.8% stage A → 59.2% stage B → 61.7% stage B at the matched 768-token/image inference resolution → 63.3% final stage C. Note these val samples were also in training, so held-out accuracy will be lower; checkpoint/resolution selection used only this sanctioned set. Selfcheck on the exact submitted folder: startup 82.1s (limit 300s), projected 900-question evaluation 183s (limit 3600s), all response gates passed.
Serving is via serve.sh, which installs the vendored cp312 CUDA wheels offline into /tmp and serves POST /v1/chat/completions plus GET /health on one H100.