Reproduce “MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices” on OpenResearch

About this paper

Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang and 3 more

Researchers from Meituan Inc., Zhejiang University, and Dalian University of Technology introduced MobileVLM, an open multimodal vision-language model optimized for mobile and edge devices. This model achieves state-of-the-art inference speeds of 21.5 tokens/s on a Qualcomm Snapdragon 888 CPU and 65.3 tokens/s on an NVIDIA Jetson Orin GPU, while maintaining competitive multimodal capabilities with much larger models.

Key insights

• Intelligent visual token reduction, exemplified by the Lightweight Downsample Projector (LDP), is critical for achieving high inference speeds on mobile devices while preserving multimodal performance.
• Supervised Fine-tuning (SFT) of the base language model with dialogue-style data significantly improves its capabilities for downstream multimodal vision-language tasks.
• CLIP-pre-trained vision encoders consistently yield better multimodal performance compared to those pre-trained with unsupervised, classification, or grounding detection methods.

Summary by alphaXiv.