Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $9.99 of $10
- Model cost
- $13.72
- Agent time
- 6h 39m
- Steps
- 276
- Tokens in / out
- 69.3M / 260k
- Self-checks
- 3 passed of 4 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.3422
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json names deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. Setup a
architecture.json names deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. Setup and training used a snapshot_download of that same revision (logs/setup.log). The shipped weights/maple directory was copied directly from /artifacts/maple/weights with cp -a (trajectory#85 region in steps-68-91). load_maple_weights loads all 18651 backbone tensors listed in the index weight_map and raises if any are missing; train.log and mix_build.log both report 18651 tensors loaded. The shard inventories show the full Maple MoE structure.
✓only vision encoder and connector addedMapleVLM consists of the SigLIP2 so400m vision_model (weights/vision.safetensors holds only vision_model.* tensors and n
MapleVLM consists of the SigLIP2 so400m vision_model (weights/vision.safetensors holds only vision_model.* tensors and no text tower), a VisionConnector (2x2 pixel shuffle plus a two-layer MLP, fc1/fc2 in weights/connector.safetensors), and the unchanged MapleForCausalLM. Image features replace IMAGE_PAD token embeddings, and no other modules are added.
✓main network ternaryThe Maple backbone is frozen during training; only connector parameters (13.6M) go to AdamW. The shipped shards come fro
The Maple backbone is frozen during training; only connector parameters (13.6M) go to AdamW. The shipped shards come from the starting artifact unchanged. Projections remain per-row ternary BF16, and the forward pass in maple_backbone.py uses the raw stored weights via F.linear, with MLP and attention math matching upstream modeling_maple.py. test_ternary.py checks the per-row ternary contract and the sha256 values against STARTING_ARTIFACT.
✓training data provenancetrain.py and DATA_MANIFEST.md declare every source. The HF datasets are ScienceQA, A-OKVQA, ChartQA, TextVQA and OCR-VQA
train.py and DATA_MANIFEST.md declare every source. The HF datasets are ScienceQA, A-OKVQA, ChartQA, TextVQA and OCR-VQA, all loaded with split='train'. LLaVA-Instruct-150K and ShareGPT4V cap100k JSONs were fetched with hf_hub_download from public HF repos, using COCO train2017 images. The remaining data is the task-provided /task/public validation.jsonl samples, which the task explicitly allows. mix_build.log shows only train sources. The trajectory and evidence show no use of val/test splits, MMMU or inductionlabs/frontier-ml-tasks. The original setup-log JSON fetch printed MISS, but a later hf_hub_download step in the trajectory obtained the JSONs from the declared public repos.
✓actual inference uses reviewed weightsserve.sh execs maple_vlm/server.py, which builds MapleVLM from weights/maple/config.json and loads weights/maple via loa
serve.sh execs maple_vlm/server.py, which builds MapleVLM from weights/maple/config.json and loads weights/maple via load_maple_weights (all tensors are required). It loads weights/vision.safetensors and weights/connector.safetensors with strict=True. handle_chat calls model.generate with these loaded modules. There are no external API calls, and HF offline mode is set. The trajectory shows the connector file is the trained out40 connector_final.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENAI, OPENROUTER, GEMINI, claude -p and codex exec found nothing. BASE_URL a
Searches of the trajectory for ANTHROPIC, OPENAI, OPENROUTER, GEMINI, claude -p and codex exec found nothing. BASE_URL appears only in SPIN CLI documentation. HF tokens were used only for downloads, which is permitted. serve.sh and server.py make no LLM API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed6h 1m into the runtook 13m 31s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ERROR: Cannot install filelock 3.32.3 (from /submission/wheels/filelock-3.32.3-py3-none-any.whl) a
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ERROR: Cannot install filelock 3.32.3 (from /submission/wheels/filelock-3.32.3-py3-none-any.whl) and filelock 4.0.9 (from /submission/wheels/filelock-4.0.9-py3-none-any.whl) because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts [serve] venv failed, falling back to --target install: ERROR: Cannot install filelock 3.32.3 (from /submission/wheels/filelock-3.32.3-py3-none-any.whl) and filelock 4.0.9 (from /submission/wheels/filelock-4.0.9-py3-none-any.whl) because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
- 2passed6h 16m into the runtook 9m 26s
startup 84.4 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 1.6 s after a 0.1 s first question; at that rate the 900 held-out questions take 10 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed6h 27m into the runtook 11m 52s
startup 91.8 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 91.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 86.2 s after a 2.1 s first question; at that rate the 900 held-out questions take 519 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 4passed6h 40m into the runtook 12m 10s
startup 90.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 172.6 s after a 3.3 s first question; at that rate the 900 held-out questions take 1,039 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Maple-VLM: SigLIP2-SO400M/512 vision tower (frozen, text tower discarded) + trained 2×2 pixel-shuffle MLP connector (13.6M params) feeding Maple's frozen ternary backbone (all 18,651 tensors bit-identical, per-row ternary verified). Connector trained on 40K VQA/instruction samples (train splits/corpora only, no MMMU, no benchmark val/test) on an A100.
Verified: final self-check passed on H100 — startup 90s (of 300), ~1s/request serial inference, 1039s projected for 900 questions (of 3600), all replies valid OpenAI shape. Provided-sample check: 77/117 MC (in-train data, so an overfit/pipeline check, not a generalization estimate).