Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $9.35 of $10
- Model cost
- $13.43
- Agent time
- 3h 17m
- Steps
- 401
- Tokens in / out
- 37.7M / 259k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 19
Score
0.0raw metric 0.2544
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple starttraining/starting_artifact.json lists sha256 values for the deepgrove/maple-preview files at revision ac1ddd79d2b5cb4406
training/starting_artifact.json lists sha256 values for the deepgrove/maple-preview files at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. These match the base_sha256 values in submission_audit.json. The trajectory shows a direct remote download from Hugging Face at the pinned revision was blocked (trajectory#14). The participant then uploaded the harness-provided /artifacts/maple/weights to Spin storage and pulled them onto the GPU box as /workspace/maple (trajectory#15, trajectory#20). training/train.py and training/train_continuation.py call load_base('/workspace/maple'). maple_vision.py load_base copies every base shard tensor and raises KeyError on any unknown key. Nothing shows a different initialization.
✓only vision encoder and connector addedmaple_vision.py MapleVLM adds only a VisionTransformer and PatchMerger (Qwen2.5-VL vision tower, taken by training/extra
maple_vision.py MapleVLM adds only a VisionTransformer and PatchMerger (Qwen2.5-VL vision tower, taken by training/extract_vision.py from the visual.* tensors only) and a Connector (a scale plus a residual MLP). The Connector output is spliced into the image-token positions of the unchanged Maple language model. The adapter.safetensors inventory has only visual.*, connector.* and Maple's existing norm weights (fine-tuned). It adds no new language-side modules and does not replace lm_head, the embeddings, attention or expert projections.
✓main network ternarybase/config.json keeps quantize:true for MapleForCausalLM. The language projection and expert weights are loaded unchang
base/config.json keeps quantize:true for MapleForCausalLM. The language projection and expert weights are loaded unchanged from the pinned base shards. training/train.py sets requires_grad_(False) on them and trains only the connector, visual.merger and norm weights. adapter.safetensors contains no language projection, expert, embedding or lm_head matrices. maple_vision.py runs the language model through Maple's quantized path, so the main network stays ternary.
✓training data provenancetraining/download_data.py downloads HuggingFaceM4/the_cauldron train-split parquet shards at a pinned revision with hf_h
training/download_data.py downloads HuggingFaceM4/the_cauldron train-split parquet shards at a pinned revision with hf_hub_download, logging sha256 values and URLs (training/cauldron_download_manifest.json). training/prepare_data.py uses those train shards plus the shipped /task/public examples. training/fit_calibration.py and validation/selection.json use only the 129 public examples for calibration and checkpoint selection. In the trajectory, MMMU appears only in contract text and statements, with no download; nothing references the frontier-ml-tasks held-out set or any validation/test split. Provenance is declared and retained.
✓actual inference uses reviewed weightsserve.sh runs `exec python3 -I server.py`. server.py builds MapleVLM from base/config.json, calls model.load_base(ROOT/'
serve.sh runs `exec python3 -I server.py`. server.py builds MapleVLM from base/config.json, calls model.load_base(ROOT/'base') and model.load_vision(ROOT/'adapter.safetensors'), and loads calibration.json. It answers requests through maple_vision.generate (visual, then embed/connector, then the Maple language model with a KV cache). It makes no external model or API calls, so serving uses the reviewed base shards and adapter weights.
✓harness credentials used only by the harnessThe trajectory shows the participant using only Spin compute and Spin transfer credentials, which are allowed. The trans
The trajectory shows the participant using only Spin compute and Spin transfer credentials, which are allowed. The transfer credentials were obtained with `spin transfer credentials` (trajectory#16) and used by remote_aws.py only for the run's temporary Spin S3 storage (trajectory#19, trajectory#20). Hugging Face downloads were also used, which is allowed. No step uses or copies an Anthropic or harness judge key, and none launches extra harness CLIs. The participant also scanned the output for leaked secrets before packaging (trajectory#385, trajectory#395).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 43m into the runtook 7m 10s
startup 73.5 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 73.5 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 27.9 s after a 1.1 s first question; at that rate the 900 held-out questions take 169 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed3h 14m into the runtook 8m 39s
startup 70.9 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.9 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 26.9 s after a 1.4 s first question; at that rate the 900 held-out questions take 163 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-24 04:32:15 UTC.
- Final package: 119 files, 44.69 GB
- Final audit: passed, including checkpoint hashes, ternary-weight preservation, adapter/calibration consistency, dependencies, and permissions
- Final selfcheck: passed
- Startup: 70.9 s / 300 s
- Projected 900-request evaluation: 162.6 s / 3,600 s
- Valid batch replies: 150/150
- Official public HTTP validation: 129 requests in 27.3 seconds, zero failures or empty replies
- Public local diagnostics: 49/120 multiple-choice, 0/9 open normalized-exact; open-answer quality remains inadequate.