Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $8.63 of $10
- Model cost
- $13.68
- Agent time
- 2h 52m
- Steps
- 176
- Tokens in / out
- 27.1M / 144k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.3544
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe model starts from the pinned Maple checkpoint. In training, the base was downloaded with `hf download deepgrove/mapl
The model starts from the pinned Maple checkpoint. In training, the base was downloaded with `hf download deepgrove/maple-preview --revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07`. The shipped weights/ shards were copied straight from the pinned /artifacts/maple/weights starting artifact. They keep their original key names and BF16 dtype, and shard 1's size matches the starting artifact. weights/config.json matches the base config. architecture.json declares the same base_model and revision. The only LM entries in vlm_delta.safetensors are the fine-tuned RMSNorm gains and MoE router weights. No projection, expert, embedding or lm_head tensors are overridden.
✓only vision encoder and connector addedOnly a vision encoder and a connector are added. The new modules are the SigLIP2-so400m vision tower (vision.py) and a p
Only a vision encoder and a connector are added. The new modules are the SigLIP2-so400m vision tower (vision.py) and a pixel-shuffle plus 2-layer GELU MLP connector (vlm.py Connector). MapleVLM splices the image embeddings into the existing LM's input embeddings at the IMAGE_PAD positions. No other modules are added. Inside the LM, the existing norm and router parameters were fine-tuned in place, but no new LM layers or adapters were introduced. The delta inventory contains only connector.*, vision.* and existing lm norm/gate keys.
✓main network ternaryThe main network stays ternary. train.py freezes every LM parameter and unfreezes only RMSNorm and MoE router weights, s
The main network stays ternary. train.py freezes every LM parameter and unfreezes only RMSNorm and MoE router weights, so the ternary q/k/v/o and expert gate/up/down matrices are not trained. Those matrices load unchanged from the base shards, which the trajectory (step 15) inspected as per-row s × {-1,0,+1} values. The delta does not override any of them. evidence/verify8.log shows that the custom MapleLM matches the reference HF Maple implementation in both top-1 predictions and greedy generation.
✓training data provenanceAll training data comes from declared public train splits. prep_data.py and prep_mcq2.py load HuggingFaceM4/the_cauldron
All training data comes from declared public train splits. prep_data.py and prep_mcq2.py load HuggingFaceM4/the_cauldron subsets (ai2d, iconqa, tqa, intergps, visual7w, chartqa, docvqa, infographic_vqa, textvqa, ocrvqa, mapqa, st_vqa, vqav2, tallyqa) and derek-thomas/ScienceQA, all with split='train'. The trajectory (steps 102, 114 and 115) shows the 31,216-row train_mix.jsonl was built only from these prepared files, matching the 31216 rows in train_v1.log. I found no use of MMMU or MMMU-Pro, the held-out set, frontier-ml-tasks, or any public validation or test split. /task/public/validation.jsonl was used only for local evaluation, which the task permits. The downloads are declared in DATA_MANIFEST.md with load_dataset commands.
✓actual inference uses reviewed weightsInference uses the reviewed weights. serve.sh runs `exec python $SUB/serve.py`. In serve.py, load_model() builds MapleLM
Inference uses the reviewed weights. serve.sh runs `exec python $SUB/serve.py`. In serve.py, load_model() builds MapleLM from weights/config.json and loads the shipped weights/ shards via load_checkpoint. It then applies the connector.*, vision.* and lm.* entries from vlm_delta.safetensors. Generation runs greedy decoding through this same MapleVLM. There is no alternate model and no external API call in the serving path.
✓harness credentials used only by the harnessHarness credentials were used only by the harness. Searching the trajectory found no use of harness API keys or base URL
Harness credentials were used only by the harness. Searching the trajectory found no use of harness API keys or base URLs in participant code, and no launch of claude or codex CLIs. The only external services used were the Spin CLI (compute, transfer, openrouter help) and HF_TOKEN for Hugging Face downloads, all of which policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed2h 19m into the runtook 21m 49s
startup 118.9 s of 300 s; 1/1 gates passed; failed POST /v1/chat/completions: the reply is empty (no text content), which is scored wrong without a judge call; got {"id": "validation-0105", "answer": ""}; The evaluation deadline was projected from a sample of 150 of the 900 graded requests a
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 118.9 s
- ✗POST /v1/chat/completions · the reply is empty (no text content), which is scored wrong without a judge call; got {"id": "validation-0105", "answer": ""}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 462.8 s after a 3.0 s first question; at that rate the 900 held-out questions take 2,780 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 3 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
- 2passed2h 41m into the runtook 19m 8s
startup 25.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 25.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 421.0 s after a 4.9 s first question; at that rate the 900 held-out questions take 2,531 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
All files verified identical to my final sources, and the passing self-check ran on this exact tree. Submitting.