Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $38.40 of $50
- Model cost
- $2.47
- Agent time
- 7h 11m
- Steps
- 1,018
- Tokens in / out
- 100.3M / 732k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 2nd of 19
Score
0.0raw metric 0.2556
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned maple starttools/download_maple.sh downloads deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. It then
tools/download_maple.sh downloads deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. It then checks every file's sha256 against /task/public/STARTING_ARTIFACT.json. The trajectory shows that check printing 'all files verified' on the GPU box. tools/convert_checkpoint.py turns those bf16 shards into the packed ternary checkpoint, and the trajectory shows that conversion running layer by layer. The shipped weights/maple shards have Maple's structure: 24 layers, 256 experts, hidden size 2048. evidence/download_maple.log lists only some of the files, but the verification step covers all of them.
✓only vision encoder and connector addedsrc/server/engine.py builds the Maple LM from weights/maple, a frozen SigLIP2 tower from weights/vision, and a VisionCon
src/server/engine.py builds the Maple LM from weights/maple, a frozen SigLIP2 tower from weights/vision, and a VisionConnector MLP from weights/connector.pt. The connector's output is spliced into Maple's token embeddings. The connector inventory holds only pre_norm, proj.0 [2048,1152], proj.2 [2048,2048], out_norm and scale. No other modules or alternate language models are added. evidence/round3/NOTES.md and tools/train_connector.py train only the connector, with the LM and vision tower frozen.
✓main network ternarysrc/maple/ternary.py ternarizes with a threshold of 0.7·mean|w|, packs the codes at 2 bits, and stores per-row alpha. At
src/maple/ternary.py ternarizes with a threshold of 0.7·mean|w|, packs the codes at 2 bits, and stores per-row alpha. At load it rebuilds the q/k/v/o and expert gate_up/down weights as ternary codes × alpha. The weights/maple safetensors inventories show U8 packed codes with F32 alphas for every attention and expert projection. Only embeddings, lm_head, norms and the router stay in higher precision, which matches the base model's own non-ternary components.
✓training data provenanceThe training corpus comes only from HuggingFaceM4/the_cauldron train parquet shards. tools/prepare_training_data.py and
The training corpus comes only from HuggingFaceM4/the_cauldron train parquet shards. tools/prepare_training_data.py and evidence/data_manifest.json record each shard's URL and sha256, and evidence/prepare_training_data.log and evidence/prepare_training_data_2.log show which subsets were kept. Round 3 reuses a multiple-choice subset of the round 2 corpus (evidence/round3/prep_mc.py) and downloads nothing new. Selection used the allowed /task/public validation examples plus a holdout taken from the Cauldron train split. Neither the shipped files nor the trajectory show MMMU data, the frontier-ml-tasks held-out set, or any validation/test split of a dataset. The trajectory shows a few early exploration parquets being deleted before data preparation.
✓actual inference uses reviewed weightsserve.sh runs `python -m server.app`. src/server/app.py creates an Engine, which loads weights/maple/config.json and the
serve.sh runs `python -m server.app`. src/server/app.py creates an Engine, which loads weights/maple/config.json and the packed safetensors through model.load_packed (src/maple/modeling.py → src/maple/ternary.py), plus weights/vision and weights/connector.pt. Generation and letter scoring go only through self.model and these components, with no fallback model or external API. The shipped weight inventories match the shapes the loader expects.
✓harness credentials used only by the harnessThe judge key used for validation came from `spin openrouter key create` (trajectory#493). It was stored in /tmp/judge_k
The judge key used for validation came from `spin openrouter key create` (trajectory#493). It was stored in /tmp/judge_key.txt and used only against the Spin proxy, api.spin.inductionlabs.com (trajectory#494, trajectory#653, trajectory#804). That is allowed Spin/OpenRouter-proxy use. The session showed no use of harness API keys or base URLs by participant code, no copying of them, and no extra harness CLIs launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed3h 19m into the runtook 3m 46s
startup 34.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 34.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 14.5 s after a 0.6 s first question; at that rate the 900 held-out questions take 88 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed5h 8m into the runtook 5m 21s
startup 51.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 51.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 38.9 s after a 0.4 s first question; at that rate the 900 held-out questions take 234 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed6h 49m into the runtook 4m 16s
startup 35.2 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 35.2 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 13.7 s after a 0.2 s first question; at that rate the 900 held-out questions take 82 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 4passed7h 13m into the runtook 3m 41s
startup 28.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 28.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 9.9 s after a 0.1 s first question; at that rate the 900 held-out questions take 60 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
What was handed in
/workspace/output/ — 149 files, 11.24 GB, digest 3b1d60df…, graded as-is.
Model. Maple 20B-A1B (deepgrove/maple-preview, rev ac1ddd79…) keeps its released ternary weights (24 layers, 256 experts top-8, 2-bit packed, unpacked on GPU at load). A frozen SigLIP2 so400m/14-384 tower feeds a 6.6M-parameter connector (LayerNorm → Linear → GELU → Linear → LayerNorm → scale) whose 256 image tokens are spliced into Maple's own embedding sequence at the <image N> marker. No second LM, no image-to-text front end.
Training. Connector only, three rounds on the Cauldron training split (90689 → 62858 → 10018 MC samples of scienceqa/ai2d/tqa/intergps). Round 3 is the shipped connector; it takes the option-letter view from 40/120 to 49/120 on the public sample while the no-picture view stays at 26/120, so the gain is image-driven. No validation/test split of anything was used, and MMMU was never touched — only the provided /task/public/ samples, which were used for selection.
Answer rule. Log-probability of the option letter at the first answer position, nothing else. Cross-validation over the public 120 picks it alone in 5/5 folds and 162/200 random half splits; 7 alternatives were measured and rejected (option-text objective, open data mixed back in, a third epoch, PMI, per-letter calibration, cyclic permutation, image placement, other token budgets).
Serving. serve.sh unpacks the vendored cp312 wheels into /tmp/maple-lib and runs a stdlib HTTP server exposing POST /v1/chat/completions, GET /health, GET /v1/models.
Verification. Four self-checks; the last one on the exact submitted tree: startup 28.6 s of 300 s, 1/1 gates passed, 150/150 replies non-empty, projected 900-question deadline 59.6 s of 3600 s. The official validate.py against the final build: 129 items, 0 failed requests, 47/120 multiple choice correct end-to-end.
Two loose ends I closed in this final pass rather than leaving them implicit: evidence/round3/train_log_d.json turned out to be the later run E's log (the trainer writes to a fixed filename, so D's was overwritten) — it is now train_log_e.json with train_d.log as D's record and a note in evidence/round3/NOTES.md; and the connector scale, the load-test timing and the training-evidence file list in architecture.json were stale, and are now the values actually measured.