Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $174.17 of $200
- Model cost
- $46.67
- Agent time
- 7h 7m
- Steps
- 667
- Tokens in / out
- 65.1M / 785k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.3822
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple starttraining/main_run/dl_lm.sh downloads deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce50
training/main_run/dl_lm.sh downloads deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. Its log, dl_lm.log, records shard sizes that match the shipped weights/base inventory (model-00001 = 4998990664 bytes). The base shards keep the original HF key names. Both maple_vl/train.py and maple_vl/serve.py load the LM through MapleLM.load_hf from those files, and run_train.sh points --lm at the downloaded /root/maple directory. The delta checkpoint holds no LM matrices, so the start is the pinned Maple checkpoint.
✓only vision encoder and connector addedThe only new modules are the Qwen2.5-VL-7B-Instruct ViT (visual.* tensors only; its LLM bytes were never requested, per
The only new modules are the Qwen2.5-VL-7B-Instruct ViT (visual.* tensors only; its LLM bytes were never requested, per training/vision_provenance.json and vision.py export_qwen_visual_weights) and a connector (merger ln_q/fc1 reused, a new fc2, out_scale). The other trained parameters are per-row gains, norm scales and special embedding rows, all inside Maple. weights/maple_vl_delta.safetensors contains only conn/special/gain/norm/vit tensors. serve.py builds one Maple LM plus the VisionTower/Connector and loads no second language model or image-to-text model.
✓main network ternaryMaple's ternary linear weights stay frozen (requires_grad=False in maple.py). bake() in maple_vl/maple.py only multiplie
Maple's ternary linear weights stay frozen (requires_grad=False in maple.py). bake() in maple_vl/maple.py only multiplies each output row by one scalar gain, which keeps each row in {-a,0,+a}. weights/base/config.json and modeling_maple.py run F.linear directly on the stored ternary bf16 weights. training/ternary_verification_report.json, produced by maple_vl/tools/verify_ternary.py through serve.load_model, reports 18,997,248 rows, 0 non-ternary and 0 sign/zero-pattern mismatches.
✓training data provenanceEvery training corpus is declared with its HF repo, config, train split and revision in training/data_provenance/SOURCES
Every training corpus is declared with its HF repo, config, train split and revision in training/data_provenance/SOURCES_USED_IN_MAIN_RUN.md, DOWNLOAD_COMMANDS.sh and code/sources.py. All file regexes are restricted to train-*.parquet or to train-only releases, and no MMMU, MMMU-Pro or frontier-ml-tasks/maple-vlm data appears anywhere. Checkpoint selection used the 129 shipped public examples (perm_eval.py reads validation.jsonl) and proxies built from smoke training-split shards (mk_proxy.py, mk_proxy_open.py). The trajectory shows MMMU mentioned only in the task statement and in the participant's own exclusion checks; the only 'maple-vlm' hit is the task's validation script. No val/test split use was found.
✓actual inference uses reviewed weightsThe serving chain is serve.sh → maple_vl/bootstrap.py (sets HF offline) → maple_vl.serve. load_model reads weights/base
The serving chain is serve.sh → maple_vl/bootstrap.py (sets HF offline) → maple_vl.serve. load_model reads weights/base through load_hf, then weights/maple_vl_delta.safetensors through a strict load_delta with no missing or extra keys. It then bakes the gains and runs the custom Engine on that model. Image features are scattered into Maple embeddings, and the code makes no remote model calls and loads no alternative model.
✓harness credentials used only by the harnessSearching the trajectory found no launches of a claude, codex or gemini CLI, no calls to anthropic.com, and no copying o
Searching the trajectory found no launches of a claude, codex or gemini CLI, no calls to anthropic.com, and no copying or use of a harness base URL. At trajectory#654 the participant read ANTHROPIC_API_KEY only inside a leak check that grepped the output directory for the key's value (no match) and listed env names with values redacted. The key was not used or written anywhere. HF_TOKEN was used only for Hugging Face downloads (dl_lm.sh, vision.py), and Spin s5cmd was used for transfers, both of which are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 22m into the runtook 8m 52s
startup 39.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 39.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 12.7 s after a 0.1 s first question; at that rate the 900 held-out questions take 76 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed2h 41m into the runtook 16m 25s
startup 52.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 52.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 31.0 s after a 0.1 s first question; at that rate the 900 held-out questions take 186 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed7h 12m into the runtook 9m 9s
startup 44.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 44.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 16.8 s after a 0.2 s first question; at that rate the 900 held-out questions take 101 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. frontier selfcheck passed, then I ran frontier submit and frontier submit --confirm at 05:20 UTC. It reports no accuracy score, and the architecture review (which can still score the submission zero) did not run, so I don't know the held-out result.
- Startup:
/healthanswered 200 after 44 s, against the 300 s limit. - Speed: the 150 public questions took 16.8 s at concurrency 16, which projects to about 101 s for the 900 held-out questions against the 3600 s limit.
- Replies: all 150 replies parsed and were non-empty, and the server was still healthy afterwards.
- Ternary check: all 18,997,248 rows of Maple's ternary linear weights are still {−a, 0, +a}, with sign and zero patterns identical to the original checkpoint.
- Accuracy on the 129 public examples: the shipped final checkpoint scored 42.6% on multiple choice when averaged over 8 option orderings, and 38.3% in the original option order. This is a rough guide only, because those examples were also used to choose inference settings.
I left /workspace/output unchanged after the self-check copy was taken. There are no symlinks or __pycache__ folders in it.