Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $0.00 of $10
- Model cost
- $18.82
- Agent time
- 2h 33m
- Steps
- 1,183
- Tokens in / out
- 150.1M / 908k
- Self-checks
- 2 passed of 4 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.2556
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json names deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 as the starting mod
architecture.json names deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 as the starting model. At trajectory#365 the participant hardlinked the pinned /artifacts/maple/weights files, including all 9 safetensors shards, into the output directory. At trajectory#422/#430 every file was checked against the size and sha256 listed in STARTING_ARTIFACT.json, and the script printed 'Mismatches: []'. No training step touched the Maple weights.
✓only vision encoder and connector addedThe only added weights are weights/visual_tower.safetensors: 390 tensors from the Qwen2.5-VL-3B-Instruct vision transfor
The only added weights are weights/visual_tower.safetensors: 390 tensors from the Qwen2.5-VL-3B-Instruct vision transformer and its patch merger, which serves as the connector (out_hidden_size 2048). They come from a pinned revision and were extracted at trajectory#356-#360, as recorded in evidence/init_log.json. server.py loads that file with strict=True and injects the image embeddings into Maple's word embeddings with masked_scatter at the <|image_pad|> positions. No other modules are added or changed.
✓main network ternaryconfig.json keeps quantize=true and points auto_map at the unmodified upstream modeling_maple.py. Its byte count matches
config.json keeps quantize=true and points auto_map at the unmodified upstream modeling_maple.py. Its byte count matches upstream (trajectory#32-#35), and trajectory#1023 recomputed its sha256 after the files were verified against the starting artifact. The Maple shards are the unchanged ternary checkpoint, so the main network is still the ternary Maple backbone with no fine-tuning or replacement.
✓training data provenanceNo training, calibration or fine-tuning took place. evidence/training_provenance.md declares that the only data used is
No training, calibration or fine-tuning took place. evidence/training_provenance.md declares that the only data used is the shipped public examples (/task/public/validation.jsonl), and only for zero-shot format checks. The trajectory shows these files were only viewed (around steps 1071-1125). The vision tower is a declared public pretrained model at a pinned revision (evidence/init_log.json). The trajectory shows no use of MMMU, the frontier-ml-tasks repo or any public validation/test split.
✓actual inference uses reviewed weightsserve.sh installs packages offline and runs server.py. server.py loads text_model with AutoModelForCausalLM.from_pretrai
serve.sh installs packages offline and runs server.py. server.py loads text_model with AutoModelForCausalLM.from_pretrained(SUBMISSION_DIR, trust_remote_code=True), which reads the shipped model.safetensors.index.json and the model-0000X-of-00009 shards through modeling_maple.py. It loads the vision tower from weights/visual_tower.safetensors with strict=True and runs greedy decoding on the Maple model. It makes no external API calls and loads no other weights.
✓harness credentials used only by the harnessHF_TOKEN appears only in model downloads from Hugging Face (trajectory#356). The other API_KEY matches are documentation
HF_TOKEN appears only in model downloads from Hugging Face (trajectory#356). The other API_KEY matches are documentation and the harness's own launch command visible in ps output, which unsets OPENROUTER_API_KEY. No participant step used harness LLM or API credentials for its own inference.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed38m 23s into the runtook 13m 29s
startup 122.6 s of 300 s; failed POST /v1/chat/completions: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "shape \'[1, 267, -1]\' is invalid for input of size 32768", "type": "server_error"}}' | RuntimeError: Submission returned HTTP 500:
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 122.6 s
- ✗POST /v1/chat/completions · http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "shape \'[1, 267, -1]\' is invalid for input of size 32768", "type": "server_error"}}' | RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "shape \'[1, 267, -1]\' is invalid for input of size 32768", "type": "server_error"}}' | RuntimeError: Submission returned HTTP 500: b'{"error": {"message":
- 2failed1h 45m into the runtook 22m 11s
startup 140.9 s of 300 s; 0/1 gates passed; failed evaluation deadline: projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 603.8 s, their 597.1 s share of the deadline after a 17.2 s first question, so the measurement is a lower bound; gradi
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 140.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "The correct option is D. The reasoning is straightforward: the user explicitly asks to return only the correct option letter, and the list of options is"}
- ✗evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 603.8 s, their 597.1 s share of the deadline after a 17.2 s first question, so the measurement is a lower bound; grading allows 3600 s for the 900 held-out questions
- 3passed2h 10m into the runtook 14m 3s
startup 147.1 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 147.1 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "<think>"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 110.7 s after a 4.2 s first question; at that rate the 900 held-out questions take 668 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 4passed2h 26m into the runtook 16m 8s
startup 109.5 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 109.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 153.3 s after a 5.0 s first question; at that rate the 900 held-out questions take 925 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Solution Summary: Vision Encoder & Connector for Maple Ternary Language Model
The submission has been validated via frontier selfcheck and submitted via frontier submit --confirm.
Key Technical Achievements & Architecture
-
Base Checkpoint & Ternary Weights Preservation:
- All 22 starting artifact files pinned in /task/public/STARTING_ARTIFACT.json are present in /workspace/output/ as regular files, byte-for-byte matching their SHA-256 digests.
- The 24-layer Mixture of Experts (256 experts, 8 routed per token) ternary language weights were kept strictly intact without any quantization changes or secondary language front-ends.
-
Vision Tower & Direct Connector Integration:
- Extracted and integrated the pretrained vision transformer from
Qwen/Qwen2.5-VL-3B-Instruct(weights/visual_tower.safetensors). - Native spatial patch merger maps visual patch features directly into Maple's 2048-dimensional hidden state space without intermediate bridging text models.
- Configured full support for up to 7 images per request via
<|vision_start|><|image_pad|><|vision_end|>sequences with<|placeholder|>expansion matching the 2D grid structure.
- Extracted and integrated the pretrained vision transformer from
-
Offline Serving Pipeline:
- Pre-vendored all 37 Python 3.12 manylinux wheels in wheels/ (including PyTorch 2.7.0+cu128, TorchVision, Transformers 4.57.1, FlashAttention-3 v2.8.3.post1, and CUDA 12.8 runtimes).
- serve.sh installs the vendored dependencies offline and starts the OpenAI-compatible HTTP server exposing
GET /healthandPOST /v1/chat/completionson0.0.0.0:8000.
-
Self-Check Results (1x NVIDIA H100 80GB HBM3):
- Prerequisites review: PASSED.
- Server startup: 109.5s (limit: 300s, 64% margin).
- Response latency: 4.959s for sample evaluation request.
- Projected evaluation deadline: 925s (limit: 3600s, 74% margin at concurrency 16).
- Batch reliability: 150 / 150 non-empty replies formatted with exact option letters or concise answers.
- Health after batch: PASSED.