Reproduce “Mixture of Diverse Size Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Manxi Sun, Wei Liu, Jian Luan, Pengzhi Gao, Bin Wang
The Mixture of Diverse Size Experts (MoDSE) architecture introduces experts with varied computational capacities within a single MoE layer, enabling adaptive routing of tokens based on their difficulty. This approach, from Xiaomi AI Lab, improved performance on downstream NLP tasks (e.g., MMLU from 26.5 to 29.9) and achieved faster training convergence compared to homogeneous MoE baselines, while maintaining the same total parameter count and inference efficiency.
Key insights
- Language prediction involves tokens of varying inherent difficulty, suggesting that a more efficient model should adapt its computational resources to match these demands.
- Introducing diversity in expert sizes within an MoE layer allows the gating network to intelligently route tokens, assigning more capable experts to challenging predictions and smaller ones to easier tasks.
- Effective load balancing in diverse expert architectures requires both hardware-aware allocation (e.g., pairing experts on GPUs) and software-level token distribution mechanisms to maintain practical efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.