Reproduce “ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Mengting Ai, Tianxin Wei, Yifan Chen, Zhichen Zeng, Ritchie Zhao, Girish Varatkar, Bita Darvish Rouhani, Xianfeng Tang and 2 more

Researchers from UIUC, NVIDIA, Apple, and Amazon introduce ResMoE, a compression technique for Mixture-of-Experts (MoE) language models that reduces parameter count by up to 75% while maintaining model performance through Wasserstein barycenter extraction and residual compression.

Key insights

  • Experts can be effectively represented as distributions of bottleneck-1 sub-MLPs
  • Compressing residuals rather than full experts leads to better parameter efficiency
  • Unstructured pruning outperforms structured pruning for residual compression

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.