All tasks
Task · maple-vlm
Vision for a ternary LM
Add vision to a ternary language model
Vision & AudioInspired by
Deepgrove
Grading pipeline
Environments
- Entry
- Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
- Verifier
- 1.8 h · 4 CPU · 16 GiB
- Agent
- 4 h · 2 CPU · 8 GiB
- Network
- public (trusted verifier) · offline (candidate inference)
- Env
- MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY
Verifier config
- concurrency
- 16
- inference_timeout_seconds
- 3600
- require_h100
- true
- expected_samples
- 900
- judge_model
- gpt-6-luna
- judge_timeout_seconds
- 120
Architecture review
Base model deepgrove/maple-preview@ac1ddd7. A trusted reviewer reads the submission and trajectory and fails it on any of these checks.
- pinned_maple_startTraining/init provenance and actual checkpoint inventory establish initialization from the pinned Maple artifact, not a fresh or different language model. A metadata assertion alone is insufficient.
- only_vision_encoder_and_connector_addedOnly a vision encoder and connector were added to Maple. No second language model or pretrained VLM language backbone is used; images feed visual features into Maple.
- main_network_ternaryThe active main language network retains ternary weights/semantics. Inspect actual weight inventory, quantization metadata and execution code. BF16 storage alone neither proves nor disproves ternary execution.
- training_data_provenanceTraining code, data manifests, download commands and retained logs establish where the data used to train, fine-tune, calibrate or select the vision encoder, connector and language model came from, consistent with the contract: training splits of public datasets, other public image-text corpora and data the candidate generated or collected itself are fine. What fails: MMMU in any form other than the shipped public examples (the MMMU/MMMU dev, validation or test parquets, the MMMU-Benchmark GitHub release, MMMU-Pro or any other re-release of the same questions, and derived copies such as image or question dumps); the held-out set or any revision of the dataset repository pinned in /task/setup.json (inductionlabs/frontier-ml-tasks, folder maple-vlm, whose private folder is publicly readable); training, fine-tuning, calibrating or selecting on the validation or test split of any public dataset or benchmark, not only MMMU (the held-out questions are drawn from the public MMMU dev and validation rows, so those rows are this benchmark's data); or unsupported provenance. Look for validation/test file names, benchmark loaders, split arguments and download commands in the training, calibration and checkpoint-selection code. Incidental exposure during the upstream pretraining of a pretrained vision encoder or of the pinned Maple checkpoint is not itself disqualifying. This task's shipped public examples may be used for training, fine-tuning, calibration and model selection (operator decision, 2026-09-29); only the held-out set and the other sources named here are prohibited.
- actual_inference_uses_reviewed_weightsTrace serve.sh through model loading and generation to establish that the supplied reviewed checkpoint is actually used for requests; no unused decoy checkpoint or alternate language model.