Reproduce “Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads” on OpenResearch

About this paper

Grant Wilkins, Srinivasan Keshav, Richard Mortier

Researchers at the University of Cambridge developed a hybrid data center model and a workload-aware scheduler to reduce energy consumption for Large Language Model (LLM) inference workloads. By dynamically allocating tasks to heterogeneous hardware, including energy-efficient Apple Silicon and high-performance NVIDIA A100 GPUs, the approach achieved a 7.5% reduction in CPU+GPU energy consumption compared to a static baseline.

Key insights

• Apple Silicon M1 Pro systems are notably more energy-efficient for smaller LLM inference workloads, while NVIDIA A100 GPUs demonstrate better energy efficiency for larger, computationally intensive tasks.
• Increases in output token count contribute more significantly to runtime and energy consumption escalation than increases in input token count, highlighting the computational intensity of iterative token generation.
• Optimizing LLM inference energy consumption requires a nuanced, workload-aware approach that matches query characteristics (token lengths) to the specific energy-performance profiles of heterogeneous hardware accelerators.

Summary by alphaXiv.