Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $0.00 of $200
- Model cost
- $10.43
- Agent time
- 2h 12m
- Steps
- 598
- Tokens in / out
- 80.2M / 379k
- Self-checks
- 1 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
0.0raw metric 0.1967
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json declares deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The
architecture.json declares deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. The session copied the pinned artifact's safetensors shards from /artifacts/maple/weights straight into the output (trajectory#455); STARTING_ARTIFACT.json identifies that artifact (trajectory#5). No training or modification was done. The shard inventories are the original BF16 Maple tensors, and shard sizes match the original, as provenance/download_and_init.log also records.
✓only vision encoder and connector addedweights/vision.safetensors holds only visual.* tensors (the Qwen2.5-VL-3B vision blocks and patch merger). These were ra
weights/vision.safetensors holds only visual.* tensors (the Qwen2.5-VL-3B vision blocks and patch merger). These were range-downloaded from the public Qwen2.5-VL-3B-Instruct checkpoint (trajectory#400). code/modeling_maple_vlm.py builds MapleVLM as the Maple language model plus lm_head plus a Qwen2.5-VL vision transformer with a merger outputting 2048 dimensions. There is no second language model or other added component.
✓main network ternaryThe Maple shards are unmodified copies. Their stored BF16 weights are ternary-valued per row, with two unique absolute v
The Maple shards are unmodified copies. Their stored BF16 weights are ternary-valued per row, with two unique absolute values plus zero (trajectory#113, trajectory#114). code/modeling_maple_vlm.py applies plain nn.Linear to these stored values without changing them, so the language network stays ternary. Only the added vision encoder and connector are non-ternary.
✓training data provenanceNo training, fine-tuning, calibration or selection corpus was used. The Maple weights are copied unchanged, and the visi
No training, fine-tuning, calibration or selection corpus was used. The Maple weights are copied unchanged, and the vision weights are the pretrained public Qwen2.5-VL-3B-Instruct tensors (provenance/download_and_init.log, trajectory#400). The session read only /task/public/validation.jsonl, which policy allows, for smoke tests and timing (trajectory#576). No other MMMU source and no validation/test split of any other public dataset was downloaded or used.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and then execs code/server.py --weights $SUBMISSION_DIR/weights. load_mode
serve.sh installs the vendored wheels offline and then execs code/server.py --weights $SUBMISSION_DIR/weights. load_model in code/server.py builds MapleVLM from weights/config.json and loads every weights/model-*.safetensors shard plus weights/vision.safetensors. The chat completions endpoint calls MODEL.generate. There is no external API or substitute model in the serving path.
✓harness credentials used only by the harnessThe session used HF_TOKEN only for Hugging Face downloads, which policy allows (trajectory#400). The only spin activity
The session used HF_TOKEN only for Hugging Face downloads, which policy allows (trajectory#400). The only spin activity was sanctioned participant tooling: help and docs, compute listing and the OpenRouter model list (trajectory#213, trajectory#215). The SPIN_API_KEY text appears only in the tool's help output. No step used grader or judge credentials, and the server makes no external calls (code/server.py).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 22m into the runtook 15m 39s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: tx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-c
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: tx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, networkx, MarkupSafe, hf-xet, fsspec, filelock, click, charset_normalizer, annotated-types, annotated-doc, uvicorn, triton, starlette, requests, pydantic, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, jinja2, nvidia-cusolver-cu12, huggingface-hub, fastapi, torch, tokenizers, transformers, torchvision, accelerate Successfully installed MarkupSafe-3.0.4 accelerate-1.15.0 annotated-doc-0.0.5 annotated-types-0.8.0 charset_normalizer-3.5.2 click-8.5.0 fastapi-0.143.0 filelock-4.0.12 fsspec-2026.9.0 hf-xet-1.7.0 huggingface-hub-0.36.2 jinja2-3.1.6 mpmath-1.3.0 networkx-3.7 nvidia-cublas-cu12-12.8.3.14 nvidia-cuda-cupti-cu12-12.8.57 nvidia-cuda-nvrtc-cu12-12.8.61 nvidia-cuda-runtime-cu12-12.8.57 nvidia-cudnn-cu12-9.7.1.26 nvidia-cufft-cu12-11.3.3.41 nvidia-cufile-cu12-1.13.0.11 nvidia-curand-cu12-10.3.9.55 nvidia-cusolver-cu12-11.7.2.55 nvidia-cusparse-cu12-12.5.7.53 nvidia-cusparselt-cu12-0.6.3 nvidia-nccl-cu12-2.26.2 nvidia-nvjitlink-cu12-12.8.61 nvidia-nvtx-cu12-12.8.55 opentelemetry-api-1.45.1 packaging-26.3 psutil-7.2.2 pydantic-2.14.0 pydantic-core-2.50.0 pyyaml-6.0.3 regex-2026.9.29 requests-2.34.2 setuptools-84.0.0 starlette-1.7.0 sympy-1.14.0 tokenizers-0.22.2 torch-2.7.0+cu128 torchvision-0.22.0+cu128 tqdm-4.70.1 transformers-4.57.1 triton-3.3.0 typing-inspection-0.4.4 urllib3-2.8.0 uvicorn-0.54.0 Dependencies installed successfully! Starting OpenAI-compatible server on 0.0.0.0:8000... The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may pro ...[clipped]
- 2failed1h 44m into the runtook 11m 36s
startup 90.0 s of 300 s; failed POST /v1/chat/completions: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Serv
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.0 s
- ✗POST /v1/chat/completions · http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error' | RuntimeError: Submission returned HTTP 500: b'Internal Server Error'
- 3passed2h 1m into the runtook 19m 19s
startup 98.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 98.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 489.6 s after a 4.6 s first question; at that rate the 900 held-out questions take 2,942 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Summary of Work
-
Architecture & Vision Integration:
- Integrated the pretrained Qwen2.5-VL-3B-Instruct vision transformer and 2-layer patch merger connector directly into Maple's 20B-A1B ternary language model.
- The vision merger output dimension (2048) matches Maple's hidden dimension (
hidden_size: 2048). - Maple's 18,651 language backbone weights were preserved completely intact in their original ternary format adhering strictly to
backbone_contract.json. - Implemented PyTorch native
F.scaled_dot_product_attention(SDPA) with sliding window causal attention support, enabling compilation-free execution on H100 SXM without requiringnvcc.
-
Offline Serving Pipeline:
- Implemented
server.pyproviding OpenAI-compatible endpoints:GET /healthresponding with HTTP 200 upon model readiness.POST /v1/chat/completionssupporting multiple images (base64 data URLs) and prompt text, outputting option letters for multiple choice and concise answers for open questions.
- Built
serve.shto install all vendoredcp312manylinux wheels offline (PyTorch 2.7.0+cu128, TorchVision 0.22.0+cu128, Transformers 4.57.1, FastAPI, Uvicorn, Pydantic, Accelerate) and launch the server. - Engineered instant meta-device skeleton initialization with direct in-place tensor loading from safetensors, achieving a startup time of ~98.9 seconds (well within the 300s limit).
- Implemented
-
Verification & Provenance:
- Added
architecture.jsonlinkingdeepgrove/maple-previewrevisionac1ddd79d2b5cb4406f5d2bebdf95406ce505a07and provenance documentationdownload_and_init.log. - Ran
frontier selfcheckon the official H100 evaluation testbed:- Review prerequisites: Passed.
- Startup: Passed (98.9s vs. 300s requirement, 67% margin).
- POST /v1/chat/completions: Passed (4.55s latency).
- Evaluation deadline: Passed (projected 2,942s vs. 3,600s requirement, 18% margin).
- Batch responses: Passed (150/150 valid parsed replies).
- Successfully submitted via
frontier submit --confirm.
- Added