Reproduce “Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin, Jincheng Cao, Zhipeng Wang, Shayan Mohajer Hamidi, Yuanda Xu, Zhengze Zhou and 2 more
OptEP prunes routed-expert models by selecting expert groups that reconstruct original layer outputs on calibration data. It relaxes subset selection into continuous optimization, then updates retained experts’ weights with regularization. On Qwen3-30B-A3B, retaining 32 of 128 experts per layer yields 87.40% MATH-500 pass@1, above REAP’s 76.07% but below the full model’s 96.40%; results depend on domain-specific calibration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.