Reproduce “Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production” on OpenResearch

About this paper

Young Jin Kim, Rawn Henry, Raffy Fahim, Hany Hassan Awadalla

Researchers from Microsoft and NVIDIA developed an efficient inference framework for large-scale Mixture of Experts (MoE) models, extending NVIDIA's FasterTransformer with optimized GPU kernels, 4-bit/8-bit quantization, and batch pruning. This approach enabled the deployment of a 5.32 billion parameter MoE model on a single V100 GPU, achieving up to 26x inference speed-up and reducing serving costs by approximately 27% compared to traditional CPU-based distilled models, all with negligible impact on translation quality.

Key insights

• Optimized GPU kernel designs, particularly for token routing and Grouped GEMM with fused and accelerated dequantization, are critical for efficient sparse MoE model inference.
• Weight-only quantization to 4-bit or 8-bit can drastically reduce the memory footprint of MoE models and improve throughput with minimal impact on translation quality.
• Deploying a single, large, optimized MoE model on GPUs offers superior quality and is more cost-effective for multilingual machine translation compared to deploying multiple smaller, distilled models on CPUs.

Summary by alphaXiv.