Reproduce “DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts” on OpenResearch

About this paper

Yuchen Feng, Bowen Shen, Naibin Gu, Jiaxuan Zhao, Peng Fu, Zheng Lin, Weiping Wang

A method termed DIVE reconstructs dense Large Language Models (LLMs) into Mixture-of-Experts (MoE) architectures by enhancing expert diversity through domain affinity mining and pruning. The approach achieves superior language modeling perplexity and downstream task accuracy while significantly reducing retraining costs compared to previous dense-to-MoE conversion strategies.

Key insights

• Structured pruning, when applied with different calibration datasets, can effectively reveal and extract diverse, specialized functionalities from a dense LLM's FFN layers.
• By clustering tasks based on pruning-derived domain correlations, distinct and specialized experts can be constructed directly from a dense model's weights, addressing expert homogeneity.
• A targeted, two-stage retraining approach, combining dense router training with sparse LoRA-based expert fine-tuning, significantly reduces the computational cost of adapting the reconstructed MoE model.
• Freezing non-FFN components like Multi-Head Attention modules during retraining does not degrade performance and contributes to overall parameter efficiency.

Summary by alphaXiv.