Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $18.83 of $50
- Model cost
- $49.67
- Agent time
- 8h 16m
- Steps
- 632
- Tokens in / out
- 58.3M / 620k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.3278
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe language model starts from the pinned Maple checkpoint. tools/pack_maple.py builds model/maple.mpk from the provided
The language model starts from the pinned Maple checkpoint. tools/pack_maple.py builds model/maple.mpk from the provided /artifacts/maple/weights safetensors shards. It records source deepgrove/maple-preview and takes the revision from /task/public/STARTING_ARTIFACT.json (ac1ddd79…), the pinned revision. The trajectory shows the pack being rebuilt (trajectory#235) and re-uploaded to the GPU machine (trajectory#237). tools/regen_shards.py then rebuilt all 9 shards from the mpk and matched them to the STARTING_ARTIFACT sha256s, reporting 'ALL EXACT' (trajectory#285-286). The shipped evidence/regen_log.txt shows only an earlier failed attempt, so that file alone proves nothing. The trajectory confirms the bit-exact check passed with the fixed code. No other LM initialization was found.
✓only vision encoder and connector addedNothing was added besides a vision encoder and connector. Engine (vlm/engine.py) builds a SigLIP2 vision tower and a Lay
Nothing was added besides a vision encoder and connector. Engine (vlm/engine.py) builds a SigLIP2 vision tower and a LayerNorm+fc1+GELU+fc2 connector, and loads model/vlm_ckpts.pt into them. The full tensor inventory of model/vlm_ckpts.pt contains only connector.* and vision.* tensors, with no LM tensors. The Maple LM is loaded unmodified from maple.mpk, and training (tools/train_connector.py, tools/adapt_public.py) updates only vision and connector parameters against the frozen LM. There is no second LM, external API call or lookup table in the serving path.
✓main network ternaryThe main network stays ternary. vlm/mpk.py pack_ternary_codes refuses any row that is not exactly {-alpha, 0, +alpha} an
The main network stays ternary. vlm/mpk.py pack_ternary_codes refuses any row that is not exactly {-alpha, 0, +alpha} and stores 2-bit codes plus a bf16 alpha per row. MapleWeights.from_mpk (vlm/maple.py) rebuilds sign*alpha for the q/k/v/o and expert gate/up/down linears. Only the router, norms, embeddings and lm_head are dense, as in the original Maple. The bit-exact regeneration against the original shards (trajectory#286) shows the LM weights are unchanged.
✓training data provenanceThe training data comes from allowed sources. evidence/data_manifest.json and tools/prep_cauldron.py / tools/train_conne
The training data comes from allowed sources. evidence/data_manifest.json and tools/prep_cauldron.py / tools/train_connector.py use HuggingFaceM4/the_cauldron with split='train' (20 configs), loaded via load_dataset in the trajectory (trajectory#235-256, trajectory#460-494). The only other data is the task's shipped public validation.jsonl, used in tools/adapt_public.py, which operator policy allows. The trajectory has no MMMU dataset download, no frontier-ml-tasks access and no validation or test split of any public dataset. MMMU appears only in restated rules and manifest text. The provenance is declared; download logs (evidence/prep_log.txt) were not independently verified beyond the trajectory.
✓actual inference uses reviewed weightsServing uses the reviewed weights. serve.sh runs python3 -m vlm.server --root $(pwd). vlm/server.py main() loads model/c
Serving uses the reviewed weights. serve.sh runs python3 -m vlm.server --root $(pwd). vlm/server.py main() loads model/config.json, model/maple.mpk, model/siglip2.safetensors, model/vlm_ckpts.pt and model/tokenizer.json into Engine. Engine (vlm/engine.py) runs vision → connector → MapleModel with greedy decoding, and handle_chat calls ENGINE.generate. No external model or API is called at inference time. The trajectory shows the adapted checkpoint being copied into the output model dir and the output listing (trajectory#586-632).
✓harness credentials used only by the harnessNo harness-credential violation was found. All model API calls used a key from 'spin openrouter key create' (trajectory#
No harness-credential violation was found. All model API calls used a key from 'spin openrouter key create' (trajectory#371) through the Spin endpoint, which policy allows. This includes the participant's local judge proxy (tools/judge_proxy.py at trajectory#377, used at trajectory#477-547). The trajectory never references ANTHROPIC/OPENAI/OPENROUTER/GEMINI API keys or base URLs, and never launches claude/codex/gemini/qwen/grok CLIs. HF_TOKEN was copied to the rented machine only to download weights and datasets (trajectory#220), with no Hub uploads (no push_to_hub or hf upload). The file uploads were Spin compute/S3 transfers (trajectory#596).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed3h 30m into the runtook 19m 4s
startup 62.9 s of 300 s; 0/1 gates passed; failed evaluation deadline: projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 581.9 s, their 555.3 s share of the deadline after a 268.5 s first question, so the measurement is a lower bound; gradi
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 62.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "We need to answer the multiple choice letter. The user asked: \"Answer only the option letter. If the question is multiple choice. If the question is multiple choice. If the question is multiple choice. If the question is multiple choice. If the qu ...[clipped]
- ✗evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 581.9 s, their 555.3 s share of the deadline after a 268.5 s first question, so the measurement is a lower bound; grading allows 3600 s for the 900 held-out questions
- 2passed8h 10m into the runtook 14m 19s
startup 71.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 518.9 s after a 9.6 s first question; at that rate the 900 held-out questions take 3,123 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the solution:
What was built
Model — Maple (ternary 19.3B MoE LM, frozen, weights bit-exact: maple.mpk 2-bit codes + per-row bf16 alpha, regen verified against STARTING_ARTIFACT.json) + SigLIP2-so400m-naflex tower + trained 2-layer MLP connector. The language network stays ternary; only vision + connector were trained.
Training (on the rented A100):
- 5,200 steps on Cauldron train splits — 156k conversations / 283k turns across 20 configs (charts, tables, docs, science QA). Enabled by a custom input-grad-only MoE autograd Function (70ms/layer vs 365ms), length-grouped batch packing, and indices-only data prep.
- Low-lr adaptation on the 129 provided public samples (explicitly allowed).
Serving (serve.sh, vendored wheels → /tmp, offline): OpenAI-compatible /v1/chat/completions with batched lockstep decoding, option-letter-constrained answers for multiple choice, think-budget forcing to bound pathological generations, and MC reply normalization.
Results
- Judge-verified on the public 129: 45.0% (58/129) vs 34.9% before adaptation.
- Final selfcheck PASSED all gates on the grading H100: startup 71.9s/300, first POST 9.6s, deadline projection 3,123s/3,600 (13% spare), 150/150 valid replies — verified against the exact submitted digest (54 files, 16.42 GB).
A mid-run selfcheck failure (a 268s repetition loop on the 4-image question, which would have missed the 1-hour eval deadline) was diagnosed and fixed with forced think-close + constrained letters + a padded-GEMM prefill path, then re-validated before submitting.