Reproduce “Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaoyu Wang, Yizhuo Liang, Jaeyong Song, Chong Li, Seo Jin Park
Moebius is a system that enables Mixture-of-Expert (MoE) large language models to dynamically switch between Expert Parallelism (EP) and Tensor Parallelism (TP) at runtime. This adaptive approach maintains optimal performance across variable workloads, achieving up to 1.25x speedup for reinforcement learning rollouts and reducing peak time-to-first-token by 3x during bursty online serving, with switches completing in 215 –434 ms.
Key insights
- A parallelism switch can be conceptualized as an ownership transfer of data slices rather than a complete model reload, as EP and TP represent different layouts of the same underlying model and data.
- Preserving CUDA graphs across parallelism switches is critical for performance, which is achieved by ensuring that tensors for both modes reside at fixed, predetermined device-side memory addresses.
- Optimized, fused direct-transfer kernels for data movement between GPUs are essential for efficient resharing of large expert weights and scattered KV cache pages, significantly outperforming generic NCCL collectives.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.