Reproduce “LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, Venkatram Vishwanath
Researchers at Argonne National Laboratory introduce LExI, a data-free, layer-adaptive technique for Mixture-of-Experts (MoE) models that optimizes active expert allocation per layer. LExI achieves enhanced inference throughput and reduced latency with negligible accuracy loss, consistently outperforming conventional expert pruning methods.
Key insights
- Different MoE models exhibit unique and varying layer-wise sensitivities to reductions in active experts, validating the need for an adaptive rather than uniform allocation strategy.
- Traditional expert pruning primarily reduces memory footprint but often fails to provide significant inference speedups and frequently leads to substantial accuracy degradation.
- Reducing active experts in layers identified as less sensitive can optimize latency without compromising overall model performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.