Reproduce “SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexandre Muzio, Alex Sun, Churan He
SEER-MoE reduces memory in mixture-of-experts models by pruning infrequently selected experts, then fine-tunes the model to use fewer experts per token. On Mixtral 8x7B, 25% pruning loses 3.85 MMLU accuracy points, versus 6.40 for Lu et al.; the two-stage model scores 48.0% on SST5 with one active expert. These task-specific results do not establish unchanged accuracy or measured speedup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.