Reproduce “The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models” on OpenResearch

About this paper

Yan Wang, Yitao Xu, Nanhan Shen, Jinyan Su, Jimin Huang, Zining Zhu

This study reveals that Mixture-of-Experts (MoE) models, despite their design for specialization, consistently rely on a 'Standing Committee' of a few core experts that handle a disproportionate amount of routing activity across diverse domains. These committees, typically comprising 2-5 experts, capture up to 70.5% of the total routing mass, forming a domain-invariant computational backbone, and their functional suppression significantly degrades model performance.

Key insights

• MoE models exhibit a consistent structural bias towards centralized computation, with a small "Standing Committee" of experts forming a stable, domain-invariant computational backbone that handles general linguistic and reasoning tasks.
• This emergent centralization challenges the prevailing assumption of widespread expert specialization and suggests a tension with common load-balancing training objectives that aim for uniform expert utilization, potentially hindering the model's natural optimization.
• The Standing Committee experts primarily process logical framing, reasoning control, and syntactic structures, while domain-specific knowledge appears to be delegated to a periphery of less frequently activated, specialized experts.

Summary by alphaXiv.