Reproduce “Rethinking Practical and Efficient Quantization Calibration for Vision-Language Models” on OpenResearch

About this paper

Zhenhao Shang, Haizhao Jing, Guoting Wei, Haokui Zhang, Rong Xiao, Jianqing Gao, Peng Wang

Researchers from Northwestern Polytechnical University, Intellifusion, and iFLYTEK introduce the Token-level Importance-aware Layer-wise Quantization (TLQ) framework, which significantly improves Post-Training Quantization (PTQ) accuracy for Vision-Language Models (VLMs) by prioritizing tokens most sensitive to quantization error using gradient information. The method also enables memory-efficient calibration of 7B VLMs on common consumer GPUs, reducing reliance on high-end hardware.

Key insights

• Not all tokens contribute equally to a VLM's final output, and redundant visual tokens can disproportionately influence and bias conventional PTQ calibration objectives.
• Leveraging gradient information provides a principled, training-free way to identify and prioritize tokens that are truly critical to inference loss, leading to more accurate quantization scales.
• Decoupling the computational components of layer-wise calibration and distributing them across multiple GPUs is crucial for making advanced PTQ techniques feasible on more widely available hardware.

Summary by alphaXiv.