Reproduce “BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference” on OpenResearch

About this paper

Yun Wang, Lingyun Yang, Senhao Yu, Yixiao Wang, Ruixing Li, Zhixiang Wei, James Yen, Zhengwei Qi

Researchers from Shanghai Jiao Tong University developed BuddyMoE, a runtime system designed to accelerate memory-constrained Mixture-of-Experts inference by exploiting expert redundancy to gracefully handle prefetch misses. This approach substitutes unavailable experts with functionally similar ones already present in GPU memory, achieving up to 10.3% higher inference throughput with a minimal 5.4% reduction in accuracy on DeepSeek-V2-Lite.

Key insights

• Large Mixture-of-Experts models exhibit substantial functional redundancy and highly non-uniform co-activation patterns among their experts.
• Gracefully tolerating prefetch misses through intelligent expert substitution is more effective than solely relying on perfect prediction to mitigate I/O latency.
• Selecting functionally similar 'buddy' experts based on co-activation frequencies can maintain model accuracy while significantly improving inference throughput.

Summary by alphaXiv.