Reproduce “SLM-Bench: A Comprehensive Benchmark of Small Language Models on Environmental Impacts--Extended Version” on OpenResearch

About this paper

Nghiem Thanh Pham, Tung Kieu, Duc-Manh Nguyen, Son Ha Xuan, Nghia Duong-Trung, Danh Le-Phuoc

SLM-Bench introduces a comprehensive framework for evaluating Small Language Models (SLMs), assessing their performance across diverse NLP tasks while quantifying computational efficiency, energy consumption, and CO2 emissions. The benchmark systematically evaluates 15 SLMs on 23 datasets and 4 hardware configurations, highlighting varied trade-offs between accuracy, speed, and environmental impact.

Key insights

• SLMs exhibit diverse performance trade-offs, with some excelling in accuracy (e.g., Llama-3.2-1B), others in computational efficiency (e.g., GPT-Neo-1.3B), and some in energy consumption (e.g., Phi-1.5B).
• Model accuracy does not consistently correlate with model size or directly with computational and environmental efficiency, suggesting that factors like architectural design and pre-training data quality are influential.
• The framework provides a standardized, open-source approach to facilitate reproducible research and inform the development and deployment of sustainable AI applications, especially in resource-constrained environments.

Summary by alphaXiv.