Reproduce “Efficiently Editing Mixture-of-Experts Models with Compressed Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifei He, Yang Liu, Chen Liang, Hany Hassan Awadalla
Researchers from the University of Illinois Urbana-Champaign and Microsoft developed "compressed experts" to enhance the efficiency of Mixture-of-Experts (MoE) models during supervised finetuning. This approach allows MoE models to recover over 90% of the performance of full-expert configurations while reducing active parameters by over 30% and inference latency by approximately 20%.
Key insights
- Approximating the contributions of auxiliary experts with lightweight compressed embeddings is effective in retaining crucial information, preventing significant performance degradation.
- Using an element-wise product for integrating compressed expert information into the hidden state offers a computationally efficient and compatible mechanism, avoiding complex dynamic parameter modifications.
- The strategy of halving the number of active full experts and replacing the remaining half with compressed experts (e.g., k_m = k_a = k/2) strikes a practical balance between efficiency gains and performance preservation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.