Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $7.20 of $10
- Model cost
- $62.00
- Agent time
- 2h 51m
- Steps
- 74
- Tokens in / out
- 20.1M / 209k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.36
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json gives deepgrove/maple-preview at ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 as the base. In trajectory s
architecture.json gives deepgrove/maple-preview at ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 as the base. In trajectory step 21/22 (steps-19-31), the participant copied the shipped weights/maple shards straight from /artifacts/maple/weights, which is the starting artifact pinned in STARTING_ARTIFACT.json. The GPU-box snapshot_download in setup_gpu.sh also requests that exact revision. Its first attempt was rate-limited (training/logs/maple_dl.log), which download_logs.txt states. training/ternary_check.txt records that all 9 shard SHA-256s match the base revision. Neither train.py nor maple_torch.py modifies the Maple weights; maple_torch.load_maple copies them unchanged with requires_grad=False.
✓only vision encoder and connector addedServing (code/server.py) loads only three components: - Maple, via code/maple_torch.py. - The Qwen2.5-VL-3B vision tower
Serving (code/server.py) loads only three components: - Maple, via code/maple_torch.py. - The Qwen2.5-VL-3B vision tower, from the extracted visual.* safetensors, via code/vision.py. - The connector checkpoint. Its inventory holds only connector.* and merger.* tensors. The Qwen LLM embed_tokens table is used offline in training/extract_qwen.py, only to initialise the connector by ridge regression. It is neither shipped nor served. No second language model and no image-to-text front end exist.
✓main network ternaryThe language-network weights are the original bf16 shards with per-row {-α,0,+α} values. They are loaded unchanged and f
The language-network weights are the original bf16 shards with per-row {-α,0,+α} values. They are loaded unchanged and frozen (code/maple_torch.py, requires_grad=False, no requantisation), and training/train.py optimises only connector and merger parameters. training/ternary_check.txt reports 18528 ternary 2-D projections and none non-ternary, with the shard hashes matching the base. The 123 non-ternary tensors are the router, embeddings, head and norms.
✓training data provenancetraining/data_prep.py downloads only train-0000x parquet shards from the public HuggingFaceM4/the_cauldron subsets, as l
training/data_prep.py downloads only train-0000x parquet shards from the public HuggingFaceM4/the_cauldron subsets, as listed in training/data_manifest.json, with download logs retained. No validation or test splits, MMMU or frontier-ml-tasks data appear in the code or the trajectory. The LLaVA-OneVision repo was only listed and never used. Serving settings and the epoch-1 vs epoch-2 choice were selected on /task/public/validation.jsonl, which the operator allows (training/eval_local.py, trajectory#69).
✓actual inference uses reviewed weightsserve.sh sets MAPLE_DIR=weights/maple, VISION_W=weights/vision/qwen25vl3b_visual.safetensors and CONNECTOR=weights/conne
serve.sh sets MAPLE_DIR=weights/maple, VISION_W=weights/vision/qwen25vl3b_visual.safetensors and CONNECTOR=weights/connector/connector.pt, then execs code/server.py. server.py loads exactly these through load_maple, load_vision_tower (strict key check) and Connector/merger load_state_dict, then runs MapleVLM in code/vlm.py. It makes no external API calls. The trajectory at step 70 confirms the shipped code matches the code that was tested and that the connector is the epoch-1 checkpoint. The selfcheck server log at step 72 shows the 9 Maple shards being loaded.
✓harness credentials used only by the harnessThe trajectory contains no use of ANTHROPIC/OPENAI API keys or *_BASE_URL variables by participant code, and no extra cl
The trajectory contains no use of ANTHROPIC/OPENAI API keys or *_BASE_URL variables by participant code, and no extra claude or codex CLI sessions; the only API_KEY/BASE_URL hits are spin documentation text in trajectory#3. HF_TOKEN was used only for Hugging Face downloads (trajectory#10), and spin transfer credentials only for the participant's own storage (trajectory#21). Both are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 9m into the runtook 17m 23s
startup 85.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 85.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 335.9 s after a 7.0 s first question; at that rate the 900 held-out questions take 2,022 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed2h 42m into the runtook 17m
startup 96.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 96.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 341.1 s after a 28.7 s first question; at that rate the 900 held-out questions take 2,075 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The serve.sh path now works end to end here: wheels install in about 60 s, model load 12 s, server ready in 90 s total. Checking training progress and cleaning up the local test server.