Reproduce “Efficient Multimodal Large Language Models: A Survey” on OpenResearch

About this paper

Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao and 5 more

A collaborative team from Tencent Youtu Lab, SJTU, BAAI, and ECNU presents a comprehensive survey categorizing and analyzing methods for developing efficient Multimodal Large Language Models. The survey demonstrates how diverse architectural and training strategies enable models with reduced computational costs to achieve performance competitive with, or sometimes exceeding, much larger general MLLMs across 14 visual-language benchmarks.

Key insights

• Efficiency in MLLMs is achieved through a multi-faceted approach, combining architectural innovations, optimized training methodologies, and strategic data utilization, rather than solely relying on smaller base models.
• Techniques like vision token compression, Mixture-of-Experts (MoE), and Mamba architectures are crucial for effectively processing high-resolution inputs and improving inference speed while maintaining scalability.
• High-quality pre-training and instruction-tuning data, coupled with parameter-efficient transfer learning techniques, are essential for adapting MLLMs to specific tasks with reduced resource expenditure and preserving pre-trained knowledge.

Summary by alphaXiv.