Reproduce “Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng
Shanghai AI Laboratory and collaborators developed Linear-MoE, a system integrating diverse linear sequence modeling (LSM) methods with Mixture-of-Experts (MoE) architectures. This framework achieves stable GPU memory consumption and consistent training throughput for sequence lengths up to 16K, along with constant memory inference and substantial speed advantages over baselines for sequences exceeding 16K, while maintaining competitive performance on language benchmarks.
Key insights
- Integrating LSM with MoE effectively addresses the quadratic complexity of attention, resulting in stable GPU memory usage and consistent throughput during training, and constant memory inference even with very long sequences.
- Hybrid Linear-MoE architectures, which interleave LSM and standard Transformer layers, offer a beneficial trade-off, achieving better performance on recall-intensive tasks compared to pure LSM models while retaining efficiency gains.
- The unified recurrence framework for LSMs provides a versatile approach to incorporate various efficient sequence modeling techniques into a scalable MoE architecture, fostering adaptability to future advancements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.