Reproduce “Energy and Policy Considerations for Deep Learning in NLP” on OpenResearch

About this paper

Emma Strubell, Ananya Ganesh, Andrew McCallum

Researchers at the University of Massachusetts Amherst quantified the financial and environmental costs of training deep learning models in Natural Language Processing (NLP), estimating that the development of a state-of-the-art model can cost over $100,000 and produce CO2e emissions equivalent to thousands of car miles. The study highlights the substantial and often unacknowledged resource demands, proposing recommendations for more sustainable and equitable research practices.

Key insights

• The full R&D lifecycle of deep learning models in NLP incurs costs thousands of times greater than a single training run, primarily due to extensive hyperparameter tuning and experimentation.
• Neural Architecture Search (NAS) approaches, while potentially offering marginal performance gains, result in extraordinarily high financial and environmental costs, demonstrating a poor cost-benefit ratio for minor improvements.
• The substantial computational resources required for cutting-edge NLP research exacerbate existing inequalities, potentially concentrating advanced work in well-funded industry labs and hindering academic innovation.

Summary by alphaXiv.