Reproduce “Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lujun Li, Zhu Qiyuan, Jiacheng Wang, Wei Li, Hao Gu, Sirui Han, Yike Guo
Researchers from Hong Kong University of Science and Technology developed Sub-MoE, a framework that compresses Mixture-of-Expert Large Language Models by merging specialized experts in a shared low-dimensional subspace. It consistently outperformed existing expert pruning and merging methods, achieving up to 50% expert reduction with notably less performance degradation across various MoE architectures.
Key insights
- Aligning diverse experts into a shared low-dimensional subspace via SVD effectively mitigates parameter conflicts, enabling robust merging without significant knowledge loss.
- Quantifying functional similarity through expert outputs and adaptively clustering experts across layers preserves more specialized knowledge than static or architectural-based grouping.
- Incorporating expert activation frequencies into the merging process allows for weighted consolidation, prioritizing more frequently utilized expert components and maintaining overall model efficacy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.