Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $16.33 of $200
- Model cost
- $4.94
- Agent time
- 4h 9m
- Steps
- 350
- Tokens in / out
- 54.4M / 81k
- Self-checks
- 0 passed of 4 used
- Rank at $200 wallet
- 2nd of 19
Batch exceeded 3600s deadline; score is zero
Score
0.0
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json declares deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The shipped map
architecture.json declares deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The shipped maple_weights were copied directly from the task-provided /artifacts/maple/weights (trajectory#94). On the training machine, the pinned revision was fetched with snapshot_download (trajectory#339). train_vlm.py keeps Maple frozen. The safetensors inventory shows the original Maple structure: BF16 lm_head, layernorms and MoE experts.
✓only vision encoder and connector addedmodel.py loads only Maple (AutoModelForCausalLM from maple_weights), a SigLIP vision encoder (AutoModel from vision_enco
model.py loads only Maple (AutoModelForCausalLM from maple_weights), a SigLIP vision encoder (AutoModel from vision_encoder) and a 2-layer MLP connector (connector.pt, which contains linear1 [2048,1152], linear2 [2048,2048] and a LayerNorm). Image features are spliced into Maple's inputs_embeds. There is no second language model and no image-to-text front end.
✓main network ternaryMaple was frozen during training, since train_vlm.py optimizes only the connector, so its weights are unchanged from the
Maple was frozen during training, since train_vlm.py optimizes only the connector, so its weights are unchanged from the provided checkpoint. The only edits to maple_weights/modeling_maple.py and maple_weights/fa3.py are compatibility fixes: a rope_type fix (trajectory#120) and a GQA kv repeat (trajectory#121). Neither changes weight quantization or replaces the ternary layers.
✓training data provenancetrain_vlm.py streams the train splits of derek-thomas/ScienceQA, HuggingFaceM4/ChartQA, lmms-lab/TextVQA and HuggingFace
train_vlm.py streams the train splits of derek-thomas/ScienceQA, HuggingFaceM4/ChartQA, lmms-lab/TextVQA and HuggingFaceM4/A-OKVQA, and architecture.json declares the same sources. These are public sources, but no download logs were kept beyond the streaming calls, so I could not verify them. I found no use of MMMU or validation/test data. The shipped script predates a later pooling edit (trajectory#330), but that edit did not change the datasets.
✓actual inference uses reviewed weightsserve.sh runs serve.py. serve.py builds MapleVLM from model.py, which loads maple_weights, vision_encoder and connector.
serve.sh runs serve.py. serve.py builds MapleVLM from model.py, which loads maple_weights, vision_encoder and connector.pt, and calls Maple's generate with the spliced inputs_embeds. These are exactly the reviewed weight files. The trained connector was retrieved into the output directory (trajectory#346).
✓harness credentials used only by the harnessThe trajectory shows no use of the harness API_KEY or BASE_URL credentials. HF_TOKEN was used only to download the model
The trajectory shows no use of the harness API_KEY or BASE_URL credentials. HF_TOKEN was used only to download the model and datasets (trajectory#339), which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 40m into the runtook 11m 57s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ERROR: Could not find a version that satisfies the requirement typer (from transformers) (from ver
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ERROR: Could not find a version that satisfies the requirement typer (from transformers) (from versions: none) ERROR: No matching distribution found for typer Traceback (most recent call last): File "/submission/serve.py", line 4, in <module> import torch ModuleNotFoundError: No module named 'torch'
- 2failed2h 54m into the runtook 8m 1s
startup 87.8 s of 300 s; failed POST /v1/chat/completions: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'{"error": "Input type (CUDABFloat16Type) and weight type (CPUBFloat16Type) should be the same"}' | RuntimeError: Submission returned HTTP 500: b'{"error":
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 87.8 s
- ✗POST /v1/chat/completions · http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'{"error": "Input type (CUDABFloat16Type) and weight type (CPUBFloat16Type) should be the same"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "Input type (CUDABFloat16Type) and weight type (CPUBFloat16Type) should be the same"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "Input type (CUDABFloat16Type) and
- 3failed3h 6m into the runtook 26m 46s
startup 92.6 s of 300 s; 1/1 gates passed; failed batch responses: 6 of 150 replies were empty, which are wrong without a judge call; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 92.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "CB"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 280.8 s after a 5.1 s first question; at that rate the 900 held-out questions take 1,690 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 6 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
- 4failed3h 33m into the runtook 20m 18s
startup 105.0 s of 300 s; 1/1 gates passed; failed batch responses: 6 of 150 replies were empty, which are wrong without a judge call; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 105.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "CB"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 551.0 s after a 6.6 s first question; at that rate the 900 held-out questions take 3,313 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 6 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
The agent's last message before its session ended.
Submission complete.