Reproduce “GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Carson Cary, Yukai Jin, Yihao Huang, Zhankai Ye, Fangyi Li, Ting Hua, Xin Liu, Shangqian Gao

GS-MoE compresses mixture-of-experts models by sharing projections within expert groups while retaining expert-specific projections and reducing intermediate widths. It uses calibration statistics to guide grouping and allocate rank and width. On DeepSeek-MoE-16B-Base, at 60% fewer parameters, its seven-task zero-shot accuracy average is 0.50, versus 0.56 uncompressed; its WikiText perplexity is worse than D²-MoE's at this setting.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.