Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $46.06 of $50
- Model cost
- $22.73
- Agent time
- 5h 37m
- Steps
- 656
- Tokens in / out
- 62.3M / 476k
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 2nd of 19
Score
0.0raw metric 0.3911
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe trajectory shows `hf download deepgrove/maple-preview --revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 --local-di
The trajectory shows `hf download deepgrove/maple-preview --revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 --local-dir /workspace/models/maple-src`. The model download commands and the Maple download log record the same command and show all shards downloading. code/pack_maple.py reads that maple-src index. It packs the q/k/v/o and expert gate/up/down projections only when each one is exactly per-row ternary and reconstructs exactly. It copies embeddings, lm_head, router and norms unchanged. The shipped tensor shapes match Maple: 24 layers, hidden size 2048, 256 experts and a 151936 vocabulary. The language network is frozen and was never retrained.
✓only vision encoder and connector addedTwo components were added. The vision tower comes from code/extract_vision.py, which keeps only Qwen2.5-VL `visual.*` ke
Two components were added. The vision tower comes from code/extract_vision.py, which keeps only Qwen2.5-VL `visual.*` keys and explicitly drops Qwen's embed_tokens, layers and lm_head; vision/vision.safetensors contains only blocks.*, merger.* and patch_embed.*. The connector is a two-layer MLP (3584→2048→2048) in connector/connector.safetensors. code/vlm_runtime.py feeds the connector output into Maple's own token embeddings, and Maple's lm_head produces the logits. No other language model or decoder is involved.
✓main network ternaryThe model/layers/layer_XX.safetensors files store the attention and expert weights as packed 2-bit U8 codes with one BF1
The model/layers/layer_XX.safetensors files store the attention and expert weights as packed 2-bit U8 codes with one BF16 alpha per row. code/maple.py unpacks them as (codes-1)*alpha, which gives exact {-α,0,+α} values, and runs attention and MoE on those weights. Embeddings, lm_head, router and norms remain in their original precision, as copied from the base. Training updates only the connector (code/train.py: AdamW over connector.parameters(), with Maple frozen).
✓training data provenanceThe training corpus is declared in architecture.json, code/prepare_data.py, evidence/DATA_PLAN.md and evidence/data_down
The training corpus is declared in architecture.json, code/prepare_data.py, evidence/DATA_PLAN.md and evidence/data_download_commands.txt as HuggingFaceM4/the_cauldron at pinned revision 847a98a7, using the hard-coded constant SPLIT="train". Trajectory load_dataset calls also show split="train" only. code/train.py and code/prepare_mc_curriculum.py skip validation/test files and rows not marked as train, and raise an error on any MMMU source. The provided /task/public/validation.jsonl was used only for checkpoint and serving selection, which is allowed. I found no download of MMMU, frontier-ml-tasks or private data, and no evaluation split. Download logs for the Cauldron are present.
✓actual inference uses reviewed weightsserve.sh runs code/server.py with HF_HUB_OFFLINE. server.py creates MapleVLM from root/model, root/vision and root/conne
serve.sh runs code/server.py with HF_HUB_OFFLINE. server.py creates MapleVLM from root/model, root/vision and root/connector. code/vlm_runtime.py calls load_maple(model) (code/maple.py loads model/embeddings, norm and layers/layer_XX), QwenVisionEncoder.from_pretrained(vision) and VisionConnector.load(connector). Multiple-choice scoring and greedy decoding both run through Maple's lm_head. server.py makes no external model API calls.
✓harness credentials used only by the harnessThe judge key came from `spin openrouter key create` into /workspace/private/judge-session.json (trajectory#301). It was
The judge key came from `spin openrouter key create` into /workspace/private/judge-session.json (trajectory#301). It was used through Spin's scoped OpenRouter base URL, which is allowed. The claude-router process seen in the trajectory is the harness's own process, shown only in `ps` output (trajectory#303). The two Agent calls are harness-native subagents, not extra CLI sessions. I found no direct use of the ANTHROPIC, OPENAI, OPENROUTER or GEMINI keys or the harness base URL, no claude/codex/gemini CLI launches, and no HF uploads in the trajectory or shipped code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 22m into the runtook 6m 16s
startup 59.3 s of 300 s; 1/1 gates passed; failed batch responses: 4 of 150 replies were empty, which are wrong without a judge call; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 14.3 s after a 0.4 s first question; at that rate the 900 held-out questions take 86 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 4 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
- 2passed2h 2m into the runtook 3m 19s
startup 56.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 56.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 16.1 s after a 0.2 s first question; at that rate the 900 held-out questions take 97 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed5h 41m into the runtook 4m 13s
startup 48.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 48.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 42.0 s after a 0.5 s first question; at that rate the 900 held-out questions take 252 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The final self-check passed on one H100:
- Startup: 48.4 seconds, under 300.
- Projected 900-request evaluation: 252.3 seconds, under 3,600.
- All 150 responses parsed and nonempty.
- Server remained healthy.
I’m saving that report, running the last audit, and then performing the requested two-step submission.