Reproduce “When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale” on OpenResearch

About this paper

Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, Sara Hooker

This research from Cohere For AI demonstrates that Large Language Models can achieve comparable or superior performance by being pretrained on significantly smaller, carefully pruned datasets, challenging the 'more data is better' paradigm. The study finds that simple perplexity-based pruning on the 'middle' range of data effectively reduces dataset size while improving model quality and downstream generalization, highlighting the importance of data quality over sheer quantity.

Key insights

• Perplexity-based pruning, a simple and computationally inexpensive method, consistently outperformed more complex metrics like EL2N and memorization for data selection.
• Removing the 'easiest' examples (lowest perplexity, lowest EL2N, or highest memorization) from the training data often degraded model performance, suggesting that these instances are not the most valuable for pretraining.
• Using larger and cleaner-trained reference models (e.g., 52B parameters, or trained on Wikipedia) to compute pruning scores led to more effective data selection and better model performance.

Summary by alphaXiv.