Reproduce “Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts” on OpenResearch

About this paper

Naibin Gu, Zhenyu Zhang, Yuchen Feng, Yilong Chen, Peng Fu, Zheng Lin, Shuohuan Wang, Yu Sun and 3 more

Elastic MoE (EMoE) is a lightweight post-training framework that enables Mixture-of-Experts (MoE) models to adapt their performance elastically by varying the number of activated experts during inference. This allows a single model to leverage increased computational resources for higher quality, overcoming the performance degradation seen in standard MoE when activating more experts than trained for.

Key insights

• The primary limitation of existing MoE models is an "inference-time scaling wall," where increasing activated experts beyond training levels leads to performance degradation due to untrained expert collaborations.
• Effectively teaching MoE models to leverage more computational resources requires exposing them to a wider range of expert co-activation patterns during training than typically used for computation.
• A robust router that produces a clear, hierarchical ranking of experts is essential to maintain high-quality expert selection and performance across diverse inference budgets, from sparse to dense activation.

Summary by alphaXiv.