Reproduce “DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeyu Zhu, Gang Li, Peisong Wang, Zitao Mo, Minnan Pei, Zhuoran Song, Xiaoyao Liang, Jian Cheng
DALI is a workload-aware offloading framework for efficient Mixture-of-Experts (MoE) inference on local PCs, integrating dynamic expert assignment, residual-based prefetching, and a workload-aware cache replacement strategy. It achieved average speedups of 3.97x in decoding and 7.62x in prefill over existing frameworks on common MoE models like Mixtral-8x7B-Instruct and DeepSeek-V2-Lite-Chat.
Key insights
- Dynamic, workload-aware assignment of experts to heterogeneous devices can significantly improve load balancing and overall inference speed compared to static approaches.
- Refining input features for expert prediction using inter-layer residuals enhances prefetching accuracy, allowing for more effective overlap of communication and computation.
- Exploiting temporal correlations in high-workload expert activations enables a dynamic cache replacement policy to maintain high cache hit rates, reducing PCIe data transfers for MoE models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.