Reproduce “MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lulu Hu, Wenhu Xiao, Xin Chen, Xinhua Xu, Bowen Xu, Kun Li, Yongliang Tao
Researchers from Alibaba Cloud Computing developed MASQuant, a post-training quantization framework for Multimodal Large Language Models (MLLMs). It achieves near full-precision accuracy even at aggressive quantization levels like W4A8 and W4A6, significantly outperforming existing methods and preventing catastrophic performance degradation for modalities with smaller activation ranges, such as audio.
Key insights
- Modality dominance in MLLMs is a critical factor leading to "smoothing misalignment" and severe performance degradation for modalities with smaller activation ranges under unified quantization.
- Applying SVD-based whitening to input activations can effectively transform the residual weight differences between modalities into a highly compressible low-rank structure, enabling accurate compensation.
- It is possible to achieve both modality-aware adaptation and computational invariance in MLLM quantization, allowing for a single quantized weight representation during inference while maintaining high accuracy across diverse modalities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.