Reproduce “SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zihan Wang, Yuqi Wang, Lei Gong, Cheng Tang, Wenqi Lou, Teng Wang, Chao Wang, Xuehai Zhou
SeqMoE combines multi-step expert-activation prediction, deadline-aware transfer scheduling, forecast-based cache replacement, and CUDA-Graph-compatible execution to reduce the cost of running MoE models when most experts remain in host memory. Across four large MoE models, it reaches roughly 75.7–84.8% of full-load performance while keeping only about 35–45% of experts resident on the GPU, and substantially improves cache hit rates over LRU and existing offloading systems.
Key insights
- Expert routing contains sufficient temporal structure for multi-step prediction: recursive eight-step forecasts lose only an average of 3.58 percentage points of recall relative to one-step prediction.
- Prediction is useful for both prefetching and eviction; jointly considering transfer confidence, deadlines, and bandwidth is more effective than ranking transfers independently.
- Once cache misses are reduced, synchronization and graph incompatibility become important bottlenecks, making runtime integration as important as the cache policy itself.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.