Reproduce “CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling” on OpenResearch

About this paper

Jihai Zhang, Xiaoye Qu, Tong Zhu, Yu Cheng

Researchers from The Chinese University of Hong Kong and Shanghai AI Laboratory developed the Diversified Multiplet Upcycling (DMU) framework, transforming pre-trained CLIP models into computationally efficient Mixture of Experts (CLIP-MoE) that capture more diverse features. This approach achieves substantial performance gains in image-text retrieval (e.g., ~20% average improvement over base CLIP) and as a vision encoder for MLLMs (e.g., +4.2 on MM-Vet) at less than 2% of the cost of training a base CLIP from scratch.

Key insights

• Diversified Multiplet Upcycling offers a computationally efficient method for scaling CLIP's capacity, drastically reducing the cost compared to training from scratch.
• Multistage Contrastive Learning effectively compels individual experts to specialize in distinct and complementary aspects of input features, mitigating CLIP's inherent information loss.
• Integrating these specialized experts into a Mixture of Experts architecture yields richer, more discriminative visual representations, enhancing performance across various multimodal tasks.

Summary by alphaXiv.