Reproduce “ST-MoE: Designing Stable and Transferable Sparse Expert Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, William Fedus

ST-MoE presents design principles and training techniques for sparse Mixture-of-Experts (MoE) models, introducing a `router z-loss` to enhance training stability and optimizing fine-tuning protocols. The resulting ST-MoE-32B model, with 269B parameters and FLOPs comparable to a 32B dense model, achieved state-of-the-art performance across a diverse set of NLP benchmarks, including an average SuperGLUE score of 91.2.

Key insights

  • The "router z-loss" is crucial for stabilizing Mixture-of-Experts (MoE) training, especially with bfloat16 precision, by mitigating roundoff errors in exponential router functions.
  • Sparse models tend to overfit on smaller fine-tuning datasets, requiring distinct fine-tuning hyperparameters (e.g., smaller batch sizes, higher learning rates) compared to dense models for optimal transfer learning.
  • Hardware-aware design principles, such as optimizing experts per core, and architectural augmentations like Sparse-Dense stacking and multiplicative biases, can enhance both MoE model efficiency and quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.