Reproduce “Expert Routing for Communication-Efficient MoE via Finite Expert Banks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammad Reza Deylam Salehi, Ali Khalesi
This research from Nice and Paris introduces a tractable empirical framework to analyze information-theoretic properties, such as algorithmic mutual information and rate-distortion trade-offs, in sparse Mixture-of-Experts models. The method validates information-theoretic generalization bounds and characterizes expert routing efficiency within a finite expert bank, which is particularly relevant for resource-constrained AI systems.
Key insights
- The algorithmic mutual information I(S;W) monotonically increases with the data-dependence parameter (α) of the learning rule, and qualitatively tracks the generalization gap in a deep learning context.
- Information-theoretic generalization bounds, while quantitatively loose, qualitatively align with the observed generalization gap, offering a nuanced understanding of generalization under gate randomization.
- The framework successfully generates empirical rate-distortion curves, validating the ability to model and analyze the accuracy-rate trade-off in expert routing as a design proxy for resource-aware MoE inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.