Reproduce “LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiawei Hao, Zhiwei Hao, Jianyuan Guo, Li Shen, Yong Luo, Han Hu, Dan Zeng

An expert replacing paradigm, LightMoE, reduces the memory footprint of Mixture-of-Experts Large Language Models by adaptively selecting and replacing redundant experts with hierarchical parameter-efficient modules. It achieves up to 50% memory reduction, showing average performance improvements of 5.6% over prior methods at 50% compression, while maintaining inference latency.

Key insights

  • Mixture-of-Experts models exhibit substantial parameter redundancy, providing an opportunity for memory-efficient compression without significant performance loss.
  • Replacing redundant experts with parameter-efficient modules, rather than pruning or merging, offers a powerful and competitive approach for MoE model compression.
  • An annealed transition strategy during fine-tuning is crucial to mitigate performance degradation and ensure training stability when replacing original experts with compressed representations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.