Reproduce “FREESH: Fair, Resource- and Energy-Efficient Scheduling for LLM Serving on Heterogeneous GPUs” on OpenResearch

About this paper

Xuan He, Zequan Fang, Jinzhao Lian, Danny H.K. Tsang, Baosen Zhang, Yize Chen

The FREESH framework introduces a hierarchical, multi-timescale optimization approach for serving large language models (LLMs) on distributed, heterogeneous GPU clusters. It reduces energy consumption by 28.60% and carbon emissions by 45.45% while improving latency metrics and maintaining fairness through adaptive GPU provisioning, dynamic frequency scaling, and a least-laxity-first scheduling policy.

Key insights

• Coordinated optimization across slow (cluster configuration) and fast (real-time GPU control and scheduling) timescales effectively balances environmental goals, performance, and fairness in LLM serving.
• Dynamically adapting control knobs such as request partitioning, Tensor Parallelism modes, and GPU frequency scaling significantly mitigates inefficiencies from static provisioning and scheduling, adapting to stochastic LLM workloads.
• Incorporating locational marginal emissions (LMEs) and request laxity into scheduling decisions enables substantial reductions in carbon footprint and improvements in fairness, particularly under high-contention scenarios.

Summary by alphaXiv.