Reproduce “MoE-Beyond: Learning-Based Expert Activation Prediction on Edge Devices” on OpenResearch

About this paper

Nishant Gavhane, Arush Mehrotra, Rohit Chawla, Peter Proenca

Researchers at the University of Pennsylvania developed MoE-Beyond, a learning-based framework using a lightweight transformer to predict expert activations for Mixture-of-Experts (MoE) models on resource-constrained edge devices. The system achieved over a 70% GPU cache hit rate with only 10% expert capacity, representing a greater than 4-fold improvement over heuristic methods.

Key insights

• Expert activation patterns exhibit significant sparsity and reuse within a single MoE inference request, despite appearing uniformly distributed when aggregated across many requests, emphasizing the need for request-aware prediction.
• A learning-based, transformer architecture can effectively capture the dynamic and sparse nature of expert activations, outperforming heuristic approaches in prediction accuracy and cache hit rates.
• Proactive and accurate prediction of expert activations is critical for optimizing GPU VRAM utilization and minimizing data transfer latencies, making large MoE models more viable for edge deployment.

Summary by alphaXiv.