Reproduce “Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models” on OpenResearch

About this paper

Soham Poddar, Paramita Koley, Janardan Misra, Sanjay Podder, Niloy Ganguly, Saptarshi Ghosh

A systematic study benchmarks the energy consumption of large language model inference across diverse models and tasks, revealing key factors influencing energy use and offering practical guidelines for more sustainable LLM deployment. The research quantifies the impact of parameters like output length, model architecture, and batching/quantization, demonstrating that optimizing these can significantly reduce energy footprints.

Key insights

• Response time serves as a highly reliable and low-overhead proxy for estimating LLM inference energy.
• Output token length has a significantly stronger influence on inference energy consumption than input token length.
• Task complexity, when input and output lengths are constant, has a negligible impact on LLM inference energy.
• Encoder-decoder models generally consume less energy than decoder-only models of comparable parameter count.

Summary by alphaXiv.