Reproduce “MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yihao Zhao, Xin Liu, Shufan Liu, Xiang Li, Yibo Zhu, Gang Huang, Xuanzhe Liu, Xin Jin
A system developed by Peking University and ByteDance, MuxFlow, facilitates efficient and safe GPU space-sharing between latency-critical online and best-effort offline deep learning workloads in large-scale production environments. It boosted overall GPU utilization from 26% to 76% across over 20,000 GPUs while maintaining online service performance and mitigating NVIDIA MPS's error propagation issues.
Key insights
- Substantial GPU underutilization in production deep learning clusters can be effectively addressed by space-sharing between latency-critical online and best-effort offline workloads.
- While NVIDIA MPS offers hardware-level space-sharing, its production deployment requires sophisticated software mechanisms for robust performance isolation and a comprehensive, mixed error-handling approach to prevent service disruption.
- Combining dynamic SM allocation for individual GPUs with a cluster-wide scheduler based on performance prediction enables efficient utilization of shared GPU resources for offline tasks without compromising online workload latency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.