Reproduce “$ϕ$-Balancing for Mixture-of-Experts Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lizhang Chen, Jonathan Li, Qi Wang, Runlong Liao, Shuozhe Li, Chen Liang, Ni Lao, Qiang Liu
The research introduces "ϕ-balancing", a principled load-balancing framework for Mixture-of-Experts (MoE) models that optimizes population-level expert utilization by minimizing a convex potential function over the global mean routing distribution. This approach achieves more stable expert utilization, leading to improved pretraining performance and competitive downstream fine-tuning results across various MoE model scales and tasks.
Key insights
- A principled framework based on minimizing a convex potential function over the *population-level* routing distribution can effectively overcome the limitations of heuristic, batch-dependent load-balancing methods.
- Convex duality provides a robust and unbiased mechanism for estimating global expert congestion, enabling an online, history-aware load-balancing algorithm that significantly improves router stability and effectiveness.
- The choice of potential function, particularly negative Shannon entropy, is critical, as it significantly impacts performance by creating a flexible pricing mechanism that promotes balanced expert utilization and sharper domain specialization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.