Reproduce “Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanfei Yu, Xingqi Cui, Hong Zhang, Hao Wang, Hao Wang
Researchers from Stevens Institute of Technology, Rice University, University of Waterloo, and Rutgers University developed FineMoE, a system that optimizes the serving efficiency of Mixture-of-Experts (MoE) Large Language Models through fine-grained expert offloading. This approach reduced inference latency by 47% and improved the expert hit rate by 39% compared to state-of-the-art solutions, effectively balancing latency and memory usage.
Key insights
- Capturing iteration-level expert activation probability distributions, rather than just aggregated counts, provides a more accurate basis for predicting expert usage.
- Combining semantic cues from input prompts and sequential expert activation trajectories across layers enhances expert prediction accuracy for both initial and later inference steps.
- Dynamically adjusting expert prefetching and offloading decisions based on prediction confidence, using a similarity-aware threshold, enables a flexible and efficient balance between latency and memory.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.