Reproduce “Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual” on OpenResearch

About this paper

Sukrit Sriratanawilai, Jhayahgrit Thongwat, Romrawin Chumpu, Patomporn Payoungkhamdee, Sarana Nutanong, Peerat Limkonchotiwat

This research systematically investigates knowledge distillation strategies for compressing multilingual Vision-Language Models (VLMs), demonstrating that smaller student models can maintain or even surpass the multilingual retrieval capabilities of their larger teachers while achieving significant efficiency gains. The optimal student models attained comparable performance to the teacher (SigLIP2-L/16) with a 2x reduction in parameters and up to a 9.7x increase in inference speed.

Key insights

• The effectiveness of KD methods is task-dependent; Distributional Replication (DR) generally excels for retrieval tasks, while English-Control Distillation (ED) is more effective for VQA.
• Multilingual student models can be successfully distilled even from English-centric teacher models, leveraging multilingual training data to enable cross-lingual knowledge transfer.
• KD can significantly improve the consistency and clustering of multilingual latent representations and occasionally enhance performance in less-resourced languages where the teacher model might be weaker.

Summary by alphaXiv.