Reproduce “SPEED-Q: Staged Processing with Enhanced Distillation towards Efficient Low-bit On-device VLM Quantization” on OpenResearch

About this paper

Tianyu Guo, Shanwei Zhao, Shiai Zhu, Chenguang Ma

A new framework, SPEED-Q, enables efficient 2-bit and 4-bit weight-only quantization for entire Vision-Language Models (VLMs), including both vision and language components, for on-device deployment. This approach minimizes accuracy degradation to approximately 3% for 4-bit and 15% for 2-bit models, while significantly reducing model size by up to 5.7x and outperforming prior methods by up to 6x in 2-bit settings.

Key insights

• The vision and language components of VLMs exhibit significantly different sensitivities to quantization, necessitating a staged processing approach to mitigate accuracy degradation.
• A distillation-enhanced quantization-aware training (QAT) approach, leveraging asymmetric clipping and adaptive self-distillation, can effectively stabilize low-bit quantization for sensitive modules and reduce reliance on large calibration datasets.
• Aggressively quantizing a well-trained, larger VLM can achieve a superior accuracy-efficiency trade-off compared to smaller, inherently lightweight models.

Summary by alphaXiv.