Reproduce “On the Spatial Structure of Mixture-of-Experts in Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel Bershatsky, Ivan Oseledets
Mixture-of-Experts (MoE) routers in Transformer models leverage positional token information alongside semantic features for expert selection, contrary to prevailing assumptions. The research provides empirical evidence of spatial correlations in expert activations, quantifies these correlations, and proposes a statistical mechanics-based framework, including a novel MEM-loss for load balancing.
Key insights
- MoE routers in Transformers exploit positional token information, such as sequence halves, in addition to semantic features for routing decisions.
- Expert activations exhibit significant spatial correlation lengths, indicating sequential dependencies, and these correlations mildly increase with MoE layer depth.
- The underlying architecture and routers primarily utilize low-frequency components of Rotary Positional Encodings (RoPE) for positional information, while high-frequency components are less relevant.
- The load balancing problem in MoE can be framed within statistical mechanics, where uneven expert activation rates correspond to high density of certain "spins," guiding the formulation of a new loss.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.