Reproduce “EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models” on OpenResearch

About this paper

Yinan Liang, Ziwei Wang, Xiuwei Xu, Jie Zhou, Jiwen Lu

An automatic pruning method for large vision-language models (LVLMs) has been introduced by researchers from Tsinghua University and Nanyang Technological University, which determines optimal sparsity configurations using only a few proxy samples. This approach yields pruned LVLMs that achieve up to a 2.16x speedup and consistently outperform state-of-the-art pruning methods across diverse multimodal question answering benchmarks.

Key insights

• Generalization ability of pruning policies for LVLMs can be effectively maximized by incorporating Structural Risk Minimization principles using only a few proxy samples, mitigating overfitting.
• Iteratively optimizing the vision projector's weights can evolve the pruning policy search space, enabling the discovery of higher-performing sparse models.
• A small dataset of 64 proxy samples and 256 token lengths is sufficient to achieve strong pruning performance for LVLMs, making the search computationally efficient.

Summary by alphaXiv.