Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $9.83 of $10
- Model cost
- $14.56
- Agent time
- 6h 26m
- Steps
- 296
- Tokens in / out
- 30.9M / 359k
- Self-checks
- 1 passed of 3 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.2467
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startIn trajectory step 125 the participant ran `python3 -m maple_vlm.pack /artifacts/maple/weights /workspace/work/packed`.
In trajectory step 125 the participant ran `python3 -m maple_vlm.pack /artifacts/maple/weights /workspace/work/packed`. That directory is the pre-staged Maple release (deepgrove/maple-preview @ ac1ddd79…). pack_artifact.log records all 9 shards being processed, then 'q_proj bit-exact OK' and 'packed=96 passthrough=123'. assemble_output.sh copies /workspace/work/packed/* into weights/packed. The packed inventories match Maple's shapes: 24 layers, hidden 2048, 256 experts, with expert tensors packed as U32 codes plus BF16 alpha. train.py loads this same artifact through MapleWeights.load_packed. No other LM initialization appears in the code or the trajectory.
✓only vision encoder and connector addedOnly two components are added: a Qwen2.5-VL-7B ViT and a 2-layer MLP connector (3584→2048→2048). visual.safetensors cont
Only two components are added: a Qwen2.5-VL-7B ViT and a 2-layer MLP connector (3584→2048→2048). visual.safetensors contains only visual.* tensors and is loaded strict=True into the stock Qwen2_5_VisionTransformerPretrainedModel. tower_final.safetensors contains only connector.fc1/fc2 weights and biases. pipeline.py asserts that the tower has no unexpected keys. Vision features replace pad-token embeddings and go straight into the Maple engine. No second language model, captioner or image-to-text front end is loaded, and server.py makes no external API calls.
✓main network ternaryThe LM is shipped as 2-bit ternary codes (U32 .packed) with per-row BF16 alpha. pack.py ternarizes with code = sign*mask
The LM is shipped as 2-bit ternary codes (U32 .packed) with per-row BF16 alpha. pack.py ternarizes with code = sign*mask + 1, and model.load_packed dequantizes as (code-1)*alpha. Attention projections and expert gate/up/down weights are all packed. Only the router, norms, embeddings and lm_head pass through, which is consistent with the release. train.py freezes the LM, and only connector parameters are trainable, so the language network stays ternary.
✓training data provenanceprep_data.py, rental_dl.py, download_log.json and architecture.json all declare the same sources: HuggingFaceM4/the_caul
prep_data.py, rental_dl.py, download_log.json and architecture.json all declare the same sources: HuggingFaceM4/the_cauldron train-*.parquet subsets, lmms-lab/LLaVA-ReCap-558K data/train-00000-of-00026.parquet, frozen Qwen2.5-VL-7B visual weights, and the /task/public validation examples. The public examples are explicitly allowed for training and selection, so selecting the checkpoint on those 129 samples is permitted. Every Cauldron and ReCap file downloaded is a train shard. Step 19 queried repo metadata only, and the participant downloaded only train files. No MMMU loader exists, and the trajectory contains no access to MMMU, inductionlabs/frontier-ml-tasks, or val/test splits.
✓actual inference uses reviewed weightsserve.sh runs `python3 -m maple_vlm.server` with `--maple-dir weights/packed`, `--tower weights/tower_final.safetensors`
serve.sh runs `python3 -m maple_vlm.server` with `--maple-dir weights/packed`, `--tower weights/tower_final.safetensors`, `--visual-weights weights/visual.safetensors` and `--tokenizer-dir weights/packed`. server.py builds the Pipeline from those paths. pipeline.py calls MapleWeights.load_packed, load_tower_bf16 and QwenVision, then runs the Engine in infer.py on the dequantized weights. In trajectory step 256-ish (steps-231-256), assemble_output.sh was run with BEST_TOWER=…/tower_ostep250.safetensors, which became tower_final. Later steps (284+) only changed decoding logic, not the weights.
✓harness credentials used only by the harnessI searched the whole trajectory for ANTHROPIC, API_KEY, BASE_URL, codex and claude CLI use. The only matches were the Sp
I searched the whole trajectory for ANTHROPIC, API_KEY, BASE_URL, codex and claude CLI use. The only matches were the Spin documentation text. Remote compute and file transfer went through the spin compute CLI and the Spin wallet S3. HF_TOKEN was used only for read-only metadata queries and downloads, and nothing in the trajectory shows an upload or push_to_hub. The shipped code makes no model API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed5h 49m into the runtook 8m 13s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: nvidia_cuda_nvrtc_cu12, nvidia_cuda_cupti_cu12, nvidia_cublas_cu12, networkx, markupsafe, jinja2,
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: nvidia_cuda_nvrtc_cu12, nvidia_cuda_cupti_cu12, nvidia_cublas_cu12, networkx, markupsafe, jinja2, huggingface_hub, fsspec, filelock, charset_normalizer Successfully installed charset_normalizer-3.5.2 filelock-4.0.12 fsspec-2026.9.0 huggingface_hub-1.33.0 jinja2-3.1.6 markupsafe-3.0.4 mpmath-1.3.0 networkx-3.7 nvidia_cublas_cu12-12.8.4.1 nvidia_cuda_cupti_cu12-12.8.90 nvidia_cuda_nvrtc_cu12-12.8.93 nvidia_cuda_runtime_cu12-12.8.90 nvidia_cudnn_cu12-9.10.2.21 nvidia_cufft_cu12-11.3.3.83 nvidia_cufile_cu12-1.13.1.3 nvidia_curand_cu12-10.3.9.90 nvidia_cusolver_cu12-11.7.3.90 nvidia_cusparse_cu12-12.5.8.93 nvidia_cusparselt_cu12-0.7.1 nvidia_nccl_cu12-2.27.5 nvidia_nvjitlink_cu12-12.8.93 nvidia_nvshmem_cu12-3.3.20 nvidia_nvtx_cu12-12.8.90 packaging-26.3 pyyaml-6.0.3 regex-2026.9.29 requests-2.34.2 sympy-1.14.0 tokenizers-0.23.2 torch-2.9.1+cu128 tqdm-4.70.1 transformers-5.18.0 triton-3.5.1 urllib3-2.8.0 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/opt/venv/lib/python3.12/site-packages/transformers/__init__.py", line 30, in <module> from . import dependency_versions_check File "/opt/venv/lib/python3.12/site-packages/transformers/dependency_versions_check.py", line 56, in <module> require_version_core(deps[pkg]) File "/opt/venv/lib/python3.12/site-packages/transformers/utils/versions.py", line 116, in require_version_core return require_version(requirement, hint) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/utils/versions.py", line 110, in require_version _compare_versions(op, got_ver, want_ver, requirement, pkg, hint) File "/opt/venv/lib/python3.12/site-packages/transformers/utils/versions.py", line 43, in _compare_versions raise ImportError( ImportError: safet ...[clipped]
- 2failed5h 59m into the runtook 4m 25s
startup 89.0 s of 300 s; 1/1 gates passed; failed batch responses: 4 of 150 replies were empty, which are wrong without a judge call; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 89.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 27.0 s after a 0.3 s first question; at that rate the 900 held-out questions take 162 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✗batch responses · 4 of 150 replies were empty, which are wrong without a judge call
- ✓healthy after requests
- 3passed6h 29m into the runtook 8m 35s
startup 76.6 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 76.6 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 26.1 s after a 1.7 s first question; at that rate the 900 held-out questions take 158 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Preview confirms 142 files / 11.71GB, all gates passed, lease already stopped. Confirming submission.