Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $9.81 of $50
- Model cost
- $22.62
- Agent time
- 3h 8m
- Steps
- 471
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 2nd of 19
Score
0.0raw metric 0.3244
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startIn the trajectory, setup_gpu.sh calls snapshot_download for deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2be
In the trajectory, setup_gpu.sh calls snapshot_download for deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 into /tmp/work/models/maple-preview (step 67). logs/download_models.log records that download finishing. train_vlm.py and run_stage1b.py both build the model from that directory with load_maple_vlm. package_submission.py takes the base shards and config from the Maple weights in /artifacts/maple/weights and patches in only the tuned tensors. It also checks every one of those tensors against the original Maple weight_map and backbone_contract.json for shape and dtype. architecture.json declares the same base model and revision.
✓only vision encoder and connector addedmodeling_maple.py adds only a SigLIP2 so400m vision tower and a MapleVisionProjector. The projector output is scattered
modeling_maple.py adds only a SigLIP2 so400m vision tower and a MapleVisionProjector. The projector output is scattered into Maple's inputs_embeds at the image_pad tokens, and no second language model exists anywhere. The vision_modules.safetensors inventory holds only vision_tower.* and vision_projector.* tensors. The model shards keep Maple's own structure (lm_head, layers, 256 experts). Training changed Maple's existing norms, routers and attention projections, but added no new language-model modules.
✓main network ternaryThe finalized attention q/k/v/o weights are written with ternarize_bf16, which produces row-wise {-α, 0, +α} values. pac
The finalized attention q/k/v/o weights are written with ternarize_bf16, which produces row-wise {-α, 0, +α} values. package_submission.py checks every self_attn *_proj row and fails if it has more than one nonzero magnitude. The MoE expert weights are never trained, so they stay Maple's original ternary weights; train_vlm.py does not save them. At inference, MapleAttention applies STE only during training and otherwise uses the stored ternary weights directly. No inventory evidence shows a non-ternary main-network weight.
✓training data provenancedownload_datasets.py and train_vlm.py load_pools use only split="train" of declared public datasets: ScienceQA, A-OKVQA,
download_datasets.py and train_vlm.py load_pools use only split="train" of declared public datasets: ScienceQA, A-OKVQA, path-vqa, vqa-rad, LLaVA-OneVision subsets and the_cauldron subsets. The code asserts the train split and rejects any repo or config containing mmmu. architecture.json lists every corpus, and logs/data_manifest.json records them. Calibration and selection used only the provided public validation.jsonl (eval_checkpoints.py), which policy allows. Searches of the trajectory found no MMMU download, no frontier-ml-tasks or held-out access, and no validation or test splits.
✓actual inference uses reviewed weightsserve.sh runs server.py with --model-dir set to the submission directory and --vision-weights set to vision_modules.safe
serve.sh runs server.py with --model-dir set to the submission directory and --vision-weights set to vision_modules.safetensors. server.py main() calls load_maple_vlm on those paths. That function loads every shard in model.safetensors.index.json, which are the patched model-0000*-of-00009.safetensors files in the inventory, and then loads vision_modules.safetensors. Requests are answered by run_vlm_inference on that model alone. The final packaging used the stage1_step1000 checkpoint (trajectory#430).
✓harness credentials used only by the harnessThe only API key the participant used was a Spin OpenRouter proxy key created with `spin openrouter key create` (traject
The only API key the participant used was a Spin OpenRouter proxy key created with `spin openrouter key create` (trajectory step 27, in steps-18-42). It was passed as MLE_JUDGE_API_KEY to the task's validate.py, pointed at the Spin base URL (trajectory#391), which policy allows. HF_TOKEN was used only to download the model (step 67). Searches found no ANTHROPIC or GEMINI keys, no harness base URLs, no environment dumps and no Hugging Face uploads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 23m into the runtook 20m 58s
startup 62.0 s of 300 s; 1/1 gates passed; failed batch responses: 2 of 150 requests failed, timed out or returned malformed completions (http_status: 2); grading scores each as a wrong answer, and the whole run zero past 5%. First errors: ["validation-0120: failed 3 attempts: Submission ret
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 62.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 571.6 s after a 3.1 s first question; at that rate the 900 held-out questions take 3,433 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 2 of 150 requests failed, timed out or returned malformed completions (http_status: 2); grading scores each as a wrong answer, and the whole run zero past 5%. First errors: ["validation-0120: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'", "validation-0121: failed 3 attempts: Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error' | Submission returned HTTP 500: b'Internal Server Error'"]
- ✓healthy after requests
- 2passed2h 12m into the runtook 16m 15s
startup 57.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 207.9 s after a 3.9 s first question; at that rate the 900 held-out questions take 1,252 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed3h 4m into the runtook 12m 24s
startup 44.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 44.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 195.6 s after a 0.3 s first question; at that rate the 900 held-out questions take 1,174 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests