All tasks
Task · ternary-ptq
Ternary quantization
Quantize Qwen3.5-4B to ternary weights with minimal accuracy loss
EfficiencyInspired by
PrismML
PrismMLGrading pipeline
Environments
- Entry
- Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
- Verifier
- 23.8 h · 4 CPU · 16 GiB
- Agent
- 4 h · 2 CPU · 8 GiB
- Network
- public (trusted verifier) · offline (candidate inference)
- Env
- MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY
Verifier config
- concurrency
- 16
- inference_timeout_seconds
- 82800
- require_h100
- true
- verifier_lifetime_seconds
- 86400
- language_suite
- gpqa-math500-lcbv6-mmlupro-v6
- benchmark_paths
- {"gpqa_diamond":"/tests/private/benchmarks/gpqa_diamond.csv","math_500":"/tests/private/benchmarks/math500.jsonl","livecodebench_v6":"/tests/private/benchmarks/livecodebench-v6","mmlu_pro":"/tests/private/benchmarks/mmlu-pro.jsonl"}
- livecodebench
- {}
- generation_concurrency
- 16
- max_tokens
- 4096
- temperature
- 0
- request_timeout_seconds
- 300
- judge_model
- gpt-6-luna
- judge_timeout_seconds
- 120
Architecture review
Base model Qwen/Qwen3.5-4B@851bf6e. A trusted reviewer reads the submission and trajectory and fails it on any of these checks.
- pinned_qwen_startTraining/init and quantization provenance establish post-training quantization from the pinned Qwen main language model. The vision tower (model.visual.*) and MTP head (mtp.*) may be omitted or kept unchanged in full precision; neither is counted towards the ternary parameter ratio. All main language model tensors remain required. The output head may be untied (an added lm_head.weight, which may be separately trained).
- text_only_inferenceThe serving path uses the main autoregressive language model. The vision tower and MTP head may remain as unused full-precision weights, but neither participates in inference or evaluation.
- no_public_benchmark_eval_trainingTrace the datasets, splits, data-generation recipes, and training/quantization logs. Task-time training, fine-tuning, distillation, and calibration, including fitting quantization scales or weights, must not use any public benchmark test or validation split, including unlabeled inputs, rewritten examples, and synthetic or teacher-generated targets derived from them. Supplied development examples are evaluation-only. This check covers work performed for this task, not the original pretraining of the prescribed base checkpoint.
- actual_inference_uses_reviewed_weightsTrace serve.sh through loading and inference to establish actual use of the inspected checkpoint, with no alternate hidden model. Compare the serving path with the numerical audit audited_storage references and the full weight inventory. Unused training/optimizer artifacts and shared storage are permitted; unreviewed active inference weights are not.
- ternary_executionThe active inference path loads the ternary checkpoint: every tensor the manifest declares literal or packed_2bit is served as code times scale and is never replaced by its original full precision values. Tensors the manifest declares float (the audit counts them as nonternary within its 5% allowance) and the input embedding and output head, which the audit excludes from the fraction and lists in trusted_audits.ternary.excluded_parameters, may legitimately be served in full precision. The trusted numerical audit is independently mandatory.