Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $14.42 of $50
- Model cost
- $48.86
- Agent time
- 2h 55m
- Steps
- 135
- Tokens in / out
- 26.4M / 180k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.3156
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe shipped Maple shards came from the pinned starting artifact. At trajectory#74 the participant ran cp /artifacts/mapl
The shipped Maple shards came from the pinned starting artifact. At trajectory#74 the participant ran cp /artifacts/maple/weights/*.safetensors (plus the config, tokenizer and license files) into /workspace/output/weights/maple. At trajectory#3, STARTING_ARTIFACT.json pins deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The training-box download at trajectory#34-47 also used that exact --revision. The shipped weights/maple shard inventory has the standard Maple tensor keys, and shard 1 is 4,998,990,664 bytes, the size the pinned artifact lists. load_maple in code/maple_model.py loads every indexed shard with no missing keys. architecture.json says these are the unmodified base shards.
✓only vision encoder and connector addedOnly two things are added around Maple. (1) A vision encoder: weights/qwen3vl2b-vision/model.safetensors holds only mode
Only two things are added around Maple. (1) A vision encoder: weights/qwen3vl2b-vision/model.safetensors holds only model.visual.* tensors, extracted at trajectory#74 by keeping keys that start with model.visual.; code/vision.py load_visual loads only those. (2) A connector: weights/vl_connector.pt holds only the connector, three DeepStack adapters and the vision special-token embeddings. The three DeepStack adapters add image features after Maple layers 0–2, and the special-token embeddings replace the image-token rows. Both are part of the vision-to-language interface. The Qwen text embedding table was used only at training time, as a ridge-regression target to initialise the connector's linear layer. It is not shipped or loaded at serve time, and no Qwen language-model layers run. The Maple LM is frozen, and trainable_state_dict in code/maple_vl.py excludes lm.* keys.
✓main network ternaryThe main network is the unmodified Maple checkpoint, which is ternary. code/maple_model.py runs F.linear and grouped_mm
The main network is the unmodified Maple checkpoint, which is ternary. code/maple_model.py runs F.linear and grouped_mm directly on the checkpoint's bf16 weights. At load it asserts that each row of the attention q/k/v/o weights and of the first 64 expert weights takes values in {-a, 0, +a}, then freezes all parameters. Stage B (the --train_merger run) was discarded and is not shipped. The LM weights were never trained or replaced.
✓training data provenanceThe training corpora are declared in code/data.py and DATA_MANIFEST.md: train-split parquet shards of HuggingFaceM4/the_
The training corpora are declared in code/data.py and DATA_MANIFEST.md: train-split parquet shards of HuggingFaceM4/the_cauldron and of 20 subsets of lmms-lab/LLaVA-OneVision-Data. The retained download commands (trajectory#48-59, training/gpu_data_download.sh) fetch only train-0000* files. No MMMU data, no inductionlabs/frontier-ml-tasks held-out data and no val/test split appear anywhere. The participant monitored training and chose the checkpoint soup using the 129 shipped /task/public validation samples, which the operator policy explicitly allows.
✓actual inference uses reviewed weightsserve.sh sets MAPLE_DIR=weights/maple, QWEN_DIR=weights/qwen3vl2b-vision and VL_CKPT=weights/vl_connector.pt, then runs
serve.sh sets MAPLE_DIR=weights/maple, QWEN_DIR=weights/qwen3vl2b-vision and VL_CKPT=weights/vl_connector.pt, then runs code/server.py. load_model in server.py calls load_maple and load_visual, builds MapleVL and loads the connector checkpoint. It asserts that no non-LM, non-vision keys are missing. Requests are then decoded greedily in code/infer.py through that same MapleVL model. There are no external model or API calls.
✓harness credentials used only by the harnessNo harness credential or launch of an extra agent CLI was found. Searches for ANTHROPIC, API keys and claude -p found on
No harness credential or launch of an extra agent CLI was found. Searches for ANTHROPIC, API keys and claude -p found only help text. At trajectory#76 the participant created their own Spin OpenRouter key and used it, via judge_env.sh at trajectory#77, as the judge key for validate.py. Spin's OpenRouter proxy is allowed. HF_TOKEN was used only for downloads, which is also allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 53m into the runtook 9m 32s
startup 85.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 85.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 9.1 s after a 1.4 s first question; at that rate the 900 held-out questions take 56 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Final pre-submit checks: verifying the checked code matches the shipped code, that every evidence file listed in architecture.json exists, and grabbing the last experiment result.