Reproduce “EMO: Frustratingly Easy Progressive Training of Extendable MoE” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Linghao Jin, Chufan Shi, Huijuan Wang, Nuan Wen, Zhengzhong Liu, Eric Xing, Xuezhe Ma
Researchers at USC-ISI and MBZUAI-IFM developed EMO, a progressive training framework that incrementally expands the expert pool of Mixture-of-Experts (MoE) models over training. EMO achieved final pretraining loss comparable to a fixed-large-expert baseline while saving 10% in GPU hours, effectively addressing the MoE efficiency paradox.
Key insights
- The full capacity of a large expert pool is not necessary in early training stages, and progressively growing expert capacity with data exposure can improve wall-clock efficiency.
- MoE experts can be conceptualized as expandable parametric memory, whose utility increases with more training data.
- Transient loss spikes after expert pool expansion are quickly recovered, indicating the model's ability to rapidly integrate newly added experts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.