Reproduce “Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xing Chen, Hengshuai Yao

Researchers identified an implicit gain miscalibration issue in fine-grained Mixture-of-Experts (MoE) models when reducing activated experts at inference time. They introduced a training-free method to decouple the activated expert count from the renormalization reference set, enabling up to 50% reduction in routed-expert compute with minimal performance degradation on language understanding and generation tasks.

Key insights

  • Performance degradation from reducing activated experts in fine-grained MoE is largely due to implicit gain miscalibration from standard renormalization, not solely capacity loss.
  • Expert selection is considerably more critical than precise expert weighting for model performance, suggesting global gain adjustments can be effective.
  • Perplexity-optimal settings for MoE compression hyperparameters can diverge from downstream task accuracy optima, cautioning against sole reliance on perplexity.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.