All tasks
Task · maple-vlm

Vision for a ternary LM

Add vision to a ternary language model

Vision & AudioInspired byDeepgrove

Grading pipeline

  1. 1Collect · snapshotHarbor collects /workspace/outputimmutable submission · digest
    ›
  2. 2Reviewautomatic · trusted verifier5 task-local checks
    ›
  3. 3Preflighttask verifier checks config and databefore GPU provisioning
    ›
  4. 4Launch · readyoffline · 1× H100 80GBbash /submission/serve.shGET /health = 200 · ≤ 300 s
    ›
  5. 5Infer900 × POST /v1/chat/completionsc=16 · ≤ 3600 s
    ›
  6. 6Judgegpt-6-lunasupported / contradicted / missing
    ›
  7. 7ScoreMMMU accuracy
    ›
  8. 8Harbor resultresult.json · reward.txt

Environments

Entry
Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
Verifier
1.8 h · 4 CPU · 16 GiB
Agent
4 h · 2 CPU · 8 GiB
Network
public (trusted verifier) · offline (candidate inference)
Env
MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY

Verifier config

concurrency
16
inference_timeout_seconds
3600
require_h100
true
expected_samples
900
judge_model
gpt-6-luna
judge_timeout_seconds
120

Architecture review

Base model deepgrove/maple-preview@ac1ddd7. A trusted reviewer reads the submission and trajectory and fails it on any of these checks.

  1. pinned_maple_start
    Training/init provenance and actual checkpoint inventory establish initialization from the pinned Maple artifact, not a fresh or different language model. A metadata assertion alone is insufficient.
  2. only_vision_encoder_and_connector_added
    Only a vision encoder and connector were added to Maple. No second language model or pretrained VLM language backbone is used; images feed visual features into Maple.
  3. main_network_ternary
    The active main language network retains ternary weights/semantics. Inspect actual weight inventory, quantization metadata and execution code. BF16 storage alone neither proves nor disproves ternary execution.
  4. training_data_provenance
    Training code, data manifests, download commands and retained logs establish where the data used to train, fine-tune, calibrate or select the vision encoder, connector and language model came from, consistent with the contract: training splits of public datasets, other public image-text corpora and data the candidate generated or collected itself are fine. What fails: MMMU in any form other than the shipped public examples (the MMMU/MMMU dev, validation or test parquets, the MMMU-Benchmark GitHub release, MMMU-Pro or any other re-release of the same questions, and derived copies such as image or question dumps); the held-out set or any revision of the dataset repository pinned in /task/setup.json (inductionlabs/frontier-ml-tasks, folder maple-vlm, whose private folder is publicly readable); training, fine-tuning, calibrating or selecting on the validation or test split of any public dataset or benchmark, not only MMMU (the held-out questions are drawn from the public MMMU dev and validation rows, so those rows are this benchmark's data); or unsupported provenance. Look for validation/test file names, benchmark loaders, split arguments and download commands in the training, calibration and checkpoint-selection code. Incidental exposure during the upstream pretraining of a pretrained vision encoder or of the pinned Maple checkpoint is not itself disqualifying. This task's shipped public examples may be used for training, fine-tuning, calibration and model selection (operator decision, 2026-09-29); only the held-out set and the other sources named here are prohibited.
  5. actual_inference_uses_reviewed_weights
    Trace serve.sh through model loading and generation to establish that the supplied reviewed checkpoint is actually used for requests; no unused decoy checkpoint or alternate language model.