Reproduce “Fine-grained MoE Load Balancing with Linear Programming” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenqi Zhao, Wenfei Wu, Linhai Song, Yuchen Xu, Yitao Yuan
FineMoE balances token loads across GPUs during Mixture-of-Experts training by routing tokens among replicas of their gate-selected experts. It reshuffles expert placement and uses a linear program to minimize the busiest GPU’s token load. Across five GPT and Mixtral pretraining configurations, it improves throughput by 36.9% on average and up to 47.6% over Megatron-LM, with DP degree greater than EP.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.