Reproduce “Revealing the Challenges of Attention-FFN Disaggregation for Modern MoE Models and Hardware Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guowei Liu, Hongming Li, Yaning Guo, Yongxi Lyu, Mo Zhou, Yi Liu, Zhaogeng Li, Yanpeng Wang
Research by Baidu Inc. rigorously analyzes Attention-FFN Disaggregation (AFD) for Mixture-of-Experts (MoE) models, revealing a "dead zone" on standard hardware due to network bottlenecks and higher penalties for load imbalance. The study identifies Superpod-class architectures and models with coarse-grained experts as conditions where AFD could be advantageous.
Key insights
- On standard cluster platforms lacking high-bandwidth scale-up networks, AFD frequently enters a network-bottlenecked "dead zone" where increasing FFN instances does not improve HFU, as the FFN stage's temporal sparsity decreases.
- AFD exhibits lower tolerance for both Data Parallelism and Expert Parallelism imbalance, as its fixed latency budget prevents reuse of released latency and its discrete scaling introduces higher quantization errors compared to continuous batch adjustments in large-scale EP.
- AFD is primarily advantageous on Superpod-class hardware with abundant interconnect bandwidth and for MoE models characterized by coarse-grained experts and lower sparsity, which counteracts current trends favoring fine-grained, highly sparse MoE designs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.