Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $34.52 of $50
- Model cost
- $19.60
- Agent time
- 11h 35m
- Steps
- 393
- Tokens in / out
- 105.2M / 375k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.2911
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe base model was downloaded with `hf download deepgrove/maple-preview --revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505
The base model was downloaded with `hf download deepgrove/maple-preview --revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07`. The shipped weights/maple shards contain the standard Maple tensor set in BF16, with no extra or renamed tensors. Both train.py and server.py load the language model with MapleForCausalLM.load_base_weights(weights/maple), which copies the indexed safetensors by name and fuses the expert weights. train.py freezes the language model. The training log shows 'loading maple base weights' before training. No other initialisation of the main network appears.
✓only vision encoder and connector addedOnly three components are added around the Maple language model: a SigLIP2 vision tower (weights/siglip2/model.safetenso
Only three components are added around the Maple language model: a SigLIP2 vision tower (weights/siglip2/model.safetensors, vision-only tensors loaded through SiglipVisionModel with assertions on missing and unexpected keys), a pixel-shuffle merge with no parameters, and an MLP connector (weights/connector.safetensors: fc1 [4096,4608] and fc2 [2048,4096] with biases). Image features are spliced into the `<|image_pad|>` embedding slots. server.py uses no other model, adapter or external service.
✓main network ternaryThe Maple weights are frozen and loaded unchanged from the pinned checkpoint, which has quantize: true in weights/maple/
The Maple weights are frozen and loaded unchanged from the pinned checkpoint, which has quantize: true in weights/maple/config.json. The fast reimplementation applies F.linear to the stored weights, which is exactly what the reference weights/maple/modeling_maple.py does. No higher-precision replacement weights, adapters or LoRA are applied to the main network, and training only updates the vision tower and connector parameters.
✓training data provenanceEvery training source is declared and taken from a train split. build_manifest.py streams split="train" from these Huggi
Every training source is declared and taken from a train split. build_manifest.py streams split="train" from these Hugging Face datasets: HuggingFaceM4/the_cauldron configs, derek-thomas/ScienceQA, HuggingFaceM4/A-OKVQA and hiyouga/geometry3k. ShareGPT4V produced 0 rows. Per-source counts are recorded in logs/build_manifest.log, logs/manifest_log.json and data_manifest.json. The extra val- rows come from the task's provided /task/public examples, which are explicitly allowed: they were copied into valpack in the session (trajectory#150) and are used by finalize_data.py, build_stage1.py and build_stage2.py. No MMMU re-release, frontier-ml-tasks repository content, or validation or test split of a public dataset was found in the code or the trajectory. The downloads are recorded only as summary logs, not full per-file download logs.
✓actual inference uses reviewed weightsserve.sh runs `exec python3 server.py --root $ROOT` with offline environment variables set. server.py builds MapleVLM fr
serve.sh runs `exec python3 server.py --root $ROOT` with offline environment variables set. server.py builds MapleVLM from three reviewed artefacts: MapleForCausalLM loaded via load_base_weights(weights/maple), VisionTower(weights/siglip2), and a Connector whose state dict comes from weights/connector.safetensors. Prefill splices the image features into the embeddings. Answers come either from the argmax over multiple-choice letter logits or from greedy decoding, both using these weights only. No alternative model or remote call is present.
✓harness credentials used only by the harnessNo trajectory step uses the harness's own API credentials; searches for OPENROUTER, API_KEY, BASE_URL and anthropic foun
No trajectory step uses the harness's own API credentials; searches for OPENROUTER, API_KEY, BASE_URL and anthropic found nothing. The only credential handling is HF_TOKEN: it was uploaded to the Spin machine (trajectory#70) and read by build_manifest.py for Hugging Face downloads. The policy explicitly allows this.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed10h 59m into the runtook 13m 27s
startup 244.5 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 244.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 11.8 s after a 4.6 s first question; at that rate the 900 held-out questions take 75 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 16:10 UTC.
Summary
Deliverable in /workspace/output/ (45 GB, 107 files): Maple VLM — frozen ternary Maple backbone (bit-identical weights) + frozen SigLIP2-SO400M-512 encoder + trained MLP connector (2×2 merge, 256 tokens/image), served via serve.sh through OpenAI-compatible POST /v1/chat/completions. No extra LM, no image-to-text frontend. Selfcheck passed (startup 244.5s/300s, gates 1/1).
Accuracy: 49/120 MC (40.8%) on the provided validation samples (stage-2; stage-1 was 48/120). Honest assessment: vision contributes weakly — text-only baseline is also ~40%, and image-swap tests show low influence. The connector learned transcription/format but not deep visual understanding within the time available.
What happened: the initial 5.3h single-stage run silently failed — the model learned to ignore the random-init connector's noise (vision unattended → no gradients) while loss 6→1 reflected format/priors only. Diagnosed via image-swap tests, weight-drift analysis (ViT moved 5e-5), and gradient probes. Recovered with the standard two-stage recipe: stage-1 alignment on 139k reading/caption rows (connector-only, FP32 master, no clip) → 40.0%, then stage-2 VQA tuning on 42k rows → 40.8%.
Engineering: fused grouped-GEMM MoE (exact match to reference), SDPA attention (no flash_attn), gradient checkpointing for 40GB-model training at ~10k tok/s, MC single-forward letter scoring for latency, stdlib-only server (always HTTP 200), 60 vendored wheels with verified offline install.
Data compliance: train splits only (Cauldron configs, ScienceQA/A-OKVQA/Geometry3K train) + provided validation samples; no val/test splits, no MMMU. Full provenance in data_manifest.json and logs/.