Reproduce “EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yize Wu, Ke Gao, Ling Li, Yanjun Wu
EasyBalance, developed by researchers at the Chinese Academy of Sciences, presents a cross-layer load balancing strategy for distributed Mixture-of-Experts (MoE) inference. This method leverages micro-batch scheduling to combine workloads from different MoE layers, consistently reducing GPU under-utilization by over 40% and lowering end-to-end inference latency across various MoE models and tasks without modifying expert-device mappings.
Key insights
- Experts from different MoE layers, residing in device memory, can be utilized for load balancing without incurring additional memory overhead, unlike traditional expert replication.
- Combining workloads from micro-batches at different layers during concurrent execution can statistically compensate for individual imbalances, reducing the effective maximum workload on any single device.
- Dynamic scheduling decisions based on current workload distributions offer instant adaptability and superior scalability compared to static, task-dependent expert-device mapping modifications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.