Reproduce “On DeepSeekMoE: Statistical Benefits of Shared Experts and Normalized Sigmoid Gating” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Huy Nguyen, Thong T. Doan, Quang Pham, Nghi D. Q. Bui, Nhat Ho, Alessandro Rinaldo
The paper analyzes how shared experts and normalized sigmoid gating affect expert estimation in sparse mixture-of-experts models. Under its assumptions, over-specified shared experts converge at approximately n^-1/4, while weakly identifiable routed experts under dense normalized sigmoid gating reach n^-1/2, up to logarithmic factors; sparse gating has no clear advantage. In SlimPajama tests, DeepSeek variants matched baseline performance in 70–80% of training steps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.