Reproduce “Load Balancing Mixture of Experts with Similarity Preserving Routers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nabil Omi, Siddhartha Sen, Ali Farhadi

Researchers from the University of Washington, Microsoft Research, and AI2 introduced SIMBAL, a novel load balancing loss for Mixture of Experts (MoE) models that preserves input similarity in routing decisions. This approach accelerates training convergence by 36%, yields lower perplexity, and fosters greater expert specialization.

Key insights

  • Uniform token distribution across experts, as promoted by traditional load balancing, can cause experts to learn redundant knowledge instead of specializing.
  • Enforcing orthogonality in the router's weight matrix can ensure that similar input tokens are routed consistently, thereby fostering expert specialization and reducing redundancy.
  • Measuring Pairwise Expert Similarity (PES) offers a computationally efficient method to quantify expert specialization and redundancy across all experts in an MoE model.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.