Reproduce “BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom and 3 more

Researchers at Cohere and the University of Oxford developed BAM (Branch-Attend-Mix), an efficient parameter upcycling method for Mixture of Experts (MoE) models that leverages both Feed-Forward Network (FFN) and attention parameters from specialized dense models. BAM consistently improved perplexity and downstream task performance over prior upcycling methods, achieving an average perplexity of 9.70 on small-scale experiments and 3.19 on large-scale models, compared to 10.72 and 3.27 respectively for the BTX baseline.

Key insights

  • Leveraging specialized attention parameters from pre-trained dense models, rather than just FFN parameters, is crucial for improving MoE model quality.
  • Soft-routing for attention experts is a critical component for achieving performance gains, promoting more stable training dynamics and maximizing the exploitation of specialized attention capabilities.
  • Parameter averaging of attention modules in prior upcycling methods like BTX can be suboptimal, as specialized knowledge is better preserved and utilized by directly mapping distinct attention modules to MoE experts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.