Reproduce “Rethinking Routing Exploration in Modern Mixture-of-Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bingcong Li, Hao Ma, Michael Muehlebach, Niao He, Yi Ma
MirrorMoE changes how a mixture-of-experts layer selects and weights routed experts: it can select low-scoring experts while assigning them negligible output weight, with suppression gradually reduced during training. At 2.4B and K=8, perplexity falls from 21.17 to 20.50. Across four model sizes, it has the lowest plotted perplexity at each tested K, but at 100M its perplexity worsens as K rises.
Key insights
- Diagnostic measures show lower expert and router representation variance with shared experts; router variance is about an order of magnitude lower in the reported comparison.
- Selecting an expert does not require giving it substantial influence: MirrorMoE can include low-probability experts while keeping their contribution small.
- Adaptive, token-dependent suppression performs better than applying a uniform reduced expert count, and the method offers little benefit without shared experts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.