Reproduce “PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units” on OpenResearch

About this paper

Yujeong Choi, Minsoo Rhu

Researchers from KAIST developed PREMA, a predictive multi-task scheduling algorithm coupled with NPU preemption mechanisms, to enhance the Quality-of-Service for deep neural network inference in multi-tenant cloud environments. The system substantially reduces tail latency for high-priority tasks to an average of 40% compared to isolated execution, alongside improvements in throughput and fairness.

Key insights

• The actual preemption latency on modern deep Neural Processing Units is negligible (microseconds, less than 2.6% of overall inference time), making NPU preemption practically viable.
• Accurate prediction of DNN inference task execution times (average estimation error of 1.6%) is crucial for effective multi-task scheduling, leading to better latency, throughput, and fairness.
• Dynamically selecting the optimal preemption mechanism (e.g., `CHECKPOINT` vs. `DRAIN`) based on predicted task remaining times consistently outperforms static preemption policies.

Summary by alphaXiv.