Reproduce “Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic” on OpenResearch

About this paper

Sachin Goyal, Pratyush Maini, Zachary C. Lipton, Aditi Raghunathan, J. Zico Kolter

A new framework of neural scaling laws demonstrates that optimal data curation is compute-aware, revealing that strategies effective at low compute become suboptimal or detrimental at higher compute budgets. The framework accounts for data heterogeneity and the diminishing utility of repeated samples, accurately predicting model performance on mixed datasets across vast compute ranges.

Key insights

• Compute-agnostic data filtering is suboptimal and can even be detrimental at high compute budgets, as the utility of a small, high-quality dataset eventually diminishes more rapidly than the benefit of more diverse, lower-quality data.
• The optimal data filtering strategy is dynamic and compute-aware, shifting from aggressive filtering at low compute to less aggressive or no filtering as compute increases.
• The utility of high-quality data decays rapidly with repetitions, sometimes becoming lower than that of initially poorer-quality data after just a few epochs.

Summary by alphaXiv.