Reproduce “Mixture-of-Experts Serving” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyi Huang, Qinpei Lou, Tao Xiao
A formal theoretical model is introduced for serving Mixture-of-Experts models, balancing service latency with the cost of dynamically reconfiguring GPU assignments. The research develops an O(sqrt(log k))-competitive online algorithm for dynamic serving and constant-factor approximations for static scenarios, concurrently demonstrating the problem's NP-hardness.
Key insights
- Efficient MoE serving necessitates balancing the conflicting objectives of minimizing service latency and reducing the overheads incurred by reconfiguring GPU assignments.
- A sqrt(log k) competitive ratio is achievable for dynamic MoE serving through online algorithms, where k represents the number of additional GPUs.
- The inherent computational complexity of static MoE serving is high, proven to be NP-hard, and unlikely to have a Fully Polynomial-Time Approximation Scheme (FPTAS).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.