Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $52.35 of $200
- Model cost
- $68.56
- Agent time
- 4h 8m
- Steps
- 161
- Tokens in / out
- 37.0M / 208k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.35
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startMaple was downloaded with snapshot_download from deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2be
Maple was downloaded with snapshot_download from deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 (trajectory around steps 30-34 and 90; logs/dl_maple.log shows the 26 files fetched). load_maple in maple_vlm/maple_core.py reads model.safetensors.index.json and copies every shipped weights/maple shard tensor unchanged. The only change is stacking the expert weights. All LM parameters have requires_grad=False and code/train.py never trains the LM, so the starting point and the served LM are the pinned Maple checkpoint.
✓only vision encoder and connector addedbuild_vlm in maple_vlm/vlm.py adds two components. The first is a Qwen3-VL-2B vision tower, initialized from Qwen/Qwen3-
build_vlm in maple_vlm/vlm.py adds two components. The first is a Qwen3-VL-2B vision tower, initialized from Qwen/Qwen3-VL-2B-Instruct using only the model.visual.* keys (code/dl_vit.py, load_vision_from_qwen3vl). The second is an MLP connector: LayerNorm, Linear, GELU, Linear, scale. Its output is masked_scatter'ed into the <|image_pad|> embedding positions of the unchanged Maple LM. weights/trainable.safetensors holds only the vision tower and connector state. No other modules, adapters or LM modifications were found.
✓main network ternaryThe Maple LM tensors are loaded unchanged from the shipped weights/maple shards and frozen; the shards were not read ele
The Maple LM tensors are loaded unchanged from the shipped weights/maple shards and frozen; the shards were not read element by element. There is no requantization, LoRA, merge or fine-tuning code for the LM. GroupedLinear's backward returns gradients only to the inputs, and train.py optimizes only the connector and vision parameters. Running the stored ternary weights in bf16 leaves their values unchanged, so the main network stays the ternary Maple model.
✓training data provenancecode/dl_data.py downloads training parquet subsets of lmms-lab/LLaVA-OneVision-Data and HuggingFaceM4/the_cauldron (docv
code/dl_data.py downloads training parquet subsets of lmms-lab/LLaVA-OneVision-Data and HuggingFaceM4/the_cauldron (docvqa, textvqa, plotqa, okvqa) using HF_TOKEN, and logs/dl_data.log plus a download manifest are retained. Both sources are declared in architecture.json and data/data_manifest.json, and maple_vlm/data.py and code/train.py read only those parquets. Validation and checkpoint selection in train.py and logs/selection.json used only the provided public validation samples, which are allowed. No MMMU (beyond the shipped public examples), frontier-ml-tasks held-out data, load_dataset calls or val/test splits appear in the code or trajectory searches.
✓actual inference uses reviewed weightsserve.sh exports MAPLE_DIR=weights/maple and WEIGHTS_DIR=weights and execs serve.py with HF_HUB_OFFLINE=1. serve.py load
serve.sh exports MAPLE_DIR=weights/maple and WEIGHTS_DIR=weights and execs serve.py with HF_HUB_OFFLINE=1. serve.py load_all calls build_vlm(weights/maple, weights/vision_config.json, weights/trainable.safetensors). Every request goes through batch_worker, then run_batch, then the maple_vlm/generation.py functions, which use only model.embed and model.lm. No external model APIs and no other checkpoints are used.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENROUTER, api_key and 'claude -p' found no matches, so the harness credentials, bas
Trajectory searches for ANTHROPIC, OPENROUTER, api_key and 'claude -p' found no matches, so the harness credentials, base URL and extra CLI sessions were never used. The participant used only allowed actions: HF_TOKEN for Hugging Face downloads, the Spin CLI and Spin storage (session.json/boto3), and environment listing.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed59m 16s into the runtook 27m 35s
startup 124.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 124.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 16.8 s after a 0.4 s first question; at that rate the 900 held-out questions take 101 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed3h 39m into the runtook 15m 3s
startup 93.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 93.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 261.1 s after a 0.7 s first question; at that rate the 900 held-out questions take 1,567 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed3h 57m into the runtook 13m 22s
startup 94.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 94.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 170.7 s after a 1.1 s first question; at that rate the 900 held-out questions take 1,025 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Naming the script profile.py shadowed the stdlib module. Renaming and rerunning.