Reproduce “SAF: Simulated Annealing Fair Scheduling for Hadoop Yarn Clusters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mahsa Ghanavatinasab, Mastaneh Bahmani, Reza Azmi
The Simulated Annealing Fair (SAF) scheduling algorithm enhances Hadoop YARN cluster performance by integrating DRF's fairness with a flexible, long-term optimization approach using Simulated Annealing, resulting in improved resource utilization and reduced job completion times.
Key insights
- A flexible, long-term optimization approach, facilitated by Simulated Annealing, can overcome the inherent fairness-performance trade-off in Hadoop YARN scheduling.
- Dynamically adjusting the annealing process based on system entropy allows the scheduler to adapt to the evolving cluster state and explore globally more efficient resource allocations.
- Meta-heuristic algorithms can introduce strategic deviations from immediate fairness to achieve better overall system performance while maintaining fairness over time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.