Reproduce “Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shwai He, Daize Dong, Liang Ding, Ang Li

Researchers from the University of Maryland and collaborators present a unified framework for compressing Mixture of Experts (MoE) models by integrating novel structured pruning techniques with individual expert compression. This approach significantly reduces the memory footprint and inference latency of large language models like Mixtral-8x7B, enabling deployment on single consumer-grade GPUs.

Key insights

  • MoE architectures exhibit significant redundancy across entire layers and blocks, particularly deeper layers, making structured pruning more effective than expert-level pruning.
  • Combining structured architectural compression (Expert Trimming) with fine-grained expert compression (Expert Slimming, especially quantization) yields superior efficiency gains compared to either method alone.
  • The proposed similarity-based metrics for identifying redundant MoE layers and blocks are robust and effectively guide the compression process without extensive calibration data tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.