Reproduce “Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenghao Lyu, Qi Fan, Fei Song, Arnab Sinha, Yanlei Diao, Wei Chen, Li Ma, Yihui Feng and 3 more
Researchers from UMass Amherst, Ecole Polytechnique, and Alibaba Group developed an intelligent resource management system for big data processing in MaxCompute, utilizing fine-grained instance-level latency prediction and a hierarchical multi-objective optimization framework. This approach delivered substantial improvements in production workloads, reducing stage latency by 37-72% and cloud cost by 43-78%.
Key insights
- Significant latency variability exists among individual instances within a big data query stage, necessitating fine-grained, instance-level performance modeling and resource assignment for optimal efficiency.
- A comprehensive multi-channel input approach, integrating diverse features like query plan topology, instance meta-data, and dynamic machine system states, is crucial for accurate instance latency prediction in complex distributed environments.
- Decomposing the complex multi-objective resource optimization problem into a hierarchical two-step process (placement then resource assignment), with efficient algorithms and clustering, enables real-time decision-making at production scale.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.