Reproduce “Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, Nipun Kwatra and 18 more
Microsoft's Singularity system enables planet-scale, preemptive, and elastic scheduling for deep learning workloads without requiring modifications to user code or frameworks. This system achieves high GPU utilization and improved fault tolerance by making all AI training jobs transparently migratable and resumable, typically introducing less than 3% overhead for execution and tens of seconds for migration.
Key insights
- Isolating device-specific state through a device-proxy allows standard CPU checkpointing tools like CRIU to be effectively applied, making deep learning jobs universally preemptible and migratable.
- Transparent elasticity can be achieved by dynamically remapping logical workers to physical devices and utilizing 'replica splicing,' with techniques such as content-checksum-based deduplication and operation squashing, to efficiently share GPU resources among time-sliced replicas.
- Critical system capabilities like preemption, migration, and elasticity can be implemented non-intrusively at the narrow interface between the CPU and AI accelerators, ensuring robustness and maintainability across various framework and CUDA versions without modifying user code.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.