Reproduce “Towards a Comprehensive Scaling Law of Mixture-of-Experts” on OpenResearch

About this paper

Guoliang Zhao, Yuhan Fu, Shuaipeng Li, Xingwu Sun, Ruobing Xie, An Wang, Weidong Han, Zhen Yang and 5 more

Tencent Hunyuan developed a comprehensive scaling law for Mixture-of-Experts (MoE) models, incorporating five key factors to predict model loss with an average error of 0.0059. The research provides empirically derived optimal configurations for MoE design, including an optimal number of active experts (G 7) and a practical optimal activated parameter ratio (5-9% for large models).

Key insights

• MoE-specific architectural parameters, namely activated model size (N_a), number of active experts (G), and ratio of shared experts (S), exhibit non-monotonic, "hook-like" relationships with model performance, implying optimal rather than simply maximal configurations.
• The theoretically optimal number of activated experts (G 6.78) and ratio of shared experts (S 0.31) were found to be largely independent of total model size, activated model size, and data size.
• The optimal activated parameter ratio (N_a/N) decreases as total model size (N) increases, indicating that larger MoE models benefit from higher sparsity, with a practical efficiency-aware optimal often ranging from 5% to 9% for very large models.

Summary by alphaXiv.