Reproduce “CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nam V. Nguyen, Huy Nguyen, Quang Pham, Van Nguyen, Savitha Ramasamy, Nhat Ho

CompeteSMoE introduces a new Sparse Mixture of Experts (SMoE) training method that employs a biologically-inspired competition mechanism for expert routing. This approach allows experts to directly compete for tokens based on their actual neural responses, demonstrating consistent performance improvements and faster convergence across visual instruction tuning and language pre-training benchmarks.

Key insights

  • Directly involving experts in the routing decision process, rather than relying on a separate router network, leads to more effective and robust Sparse Mixture of Experts (SMoE) training.
  • A biologically-inspired 'winner-take-all' competition mechanism can efficiently determine expert selection by evaluating each expert's actual neural response to a token.
  • Combining router training with distillation and a diversity loss helps manage computational overhead and prevents expert collapse, promoting specialization and improving overall model performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.