Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $5.82 of $10
- Model cost
- $23.10
- Agent time
- 2h 17m
- Steps
- 330
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 19
Score
0.0raw metric 0.3422
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe Maple copy used for training was downloaded in trajectory step 89 with snapshot_download of deepgrove/maple-preview
The Maple copy used for training was downloaded in trajectory step 89 with snapshot_download of deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The training and calibration scripts (train_vlm.py, calibrate_soup.py) load that copy from /dev/shm/work/maple. export_submission.py copied the served shards from the provided /artifacts/maple/weights, whose STARTING_ARTIFACT records the same repo and revision. In trajectory step 305 the export output checked all 18,651 backbone_contract tensors and reported them verified. architecture.json declares the same base model and revision.
✓only vision encoder and connector addedmodeling_maple_vlm.py builds MapleVLM from three parts: a SigLIP2 so400m vision tower, MapleVisionConnector (a pixel-shu
modeling_maple_vlm.py builds MapleVLM from three parts: a SigLIP2 so400m vision tower, MapleVisionConnector (a pixel-shuffle MLP), and MapleForCausalLM. The connector's output is masked_scattered into the image_pad token embeddings. The weight inventory of vlm_delta.safetensors holds only connector.* tensors (9 of them). server.py calls nothing beyond this single model: no second language model, no captioner or OCR front end, no external API.
✓main network ternaryThe served Maple shards are the original pinned shards. export_submission.py patches shards only for language_model delt
The served Maple shards are the original pinned shards. export_submission.py patches shards only for language_model delta keys, and the delta has none. The export reported that the shards still match the backbone contract. config.json sets quantize true. load_maple_vlm calls snap_ternary_weights_ after loading the delta, so TernaryLinear weights are snapped row-wise to {-α,0,+α}. Stage-2 training touched some LLM parameters, but the shipped delta, a connector soup built by calibrate_soup.py, contains none of them.
✓training data provenanceprepare_data.py and data_manifest.json download only train-split parquets: derek-thomas/ScienceQA at revision f18b0a7 an
prepare_data.py and data_manifest.json download only train-split parquets: derek-thomas/ScienceQA at revision f18b0a7 and the listed HuggingFaceM4/the_cauldron subsets at revision 847a98a. Each file has a sha256 and an hf download command, and architecture.json declares the same corpora. Selection, calibration and refinement used /task/public/validation.jsonl (packed in trajectory step 85), which is the allowed shipped public example set. Step 61 only queried dataset metadata. The trajectory shows no MMMU download, no frontier-ml-tasks held-out data, and no benchmark validation or test split being downloaded.
✓actual inference uses reviewed weightsserve.sh runs `exec python3 server.py` in offline mode. server.py main() calls load_maple_vlm with the root Maple shards
serve.sh runs `exec python3 server.py` in offline mode. server.py main() calls load_maple_vlm with the root Maple shards from model.safetensors.index.json, vision_encoder/model.safetensors and vlm_delta.safetensors. It answers every request with that model: letter logits for multiple-choice questions, and VLM_MODEL.generate for open questions. These are the same weight files covered by the inventories. Trajectory step 305 exported the calibrated soup delta into the output directory.
✓harness credentials used only by the harnessI searched the trajectory for ANTHROPIC, OPENAI, OPENROUTER, GEMINI_, API_KEY, BASE_URL, claude and codex. The only hits
I searched the trajectory for ANTHROPIC, OPENAI, OPENROUTER, GEMINI_, API_KEY, BASE_URL, claude and codex. The only hits are Spin documentation text (SPIN_API_KEY/SPIN_BASE_URL). The participant used the Spin CLI and its storage credentials, and HF_TOKEN for Hugging Face API and download calls, all of which are allowed. I found no use or copying of harness API keys or base URLs, and no extra harness CLI launches.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 26m into the runtook 16m 34s
startup 128.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 128.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 29.1 s after a 0.6 s first question; at that rate the 900 held-out questions take 175 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed2h 13m into the runtook 11m 40s
startup 84.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 73.8 s after a 0.5 s first question; at that rate the 900 held-out questions take 443 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests