Reproduce “Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining” on OpenResearch

About this paper

Anirudh Subramanyam, Yuxin Chen, Robert L. Grossman

Researchers at the University of Chicago introduce a quality-aware scaling law that explicitly incorporates data quality into the pretraining loss prediction for large language models. The framework demonstrates empirically that higher data quality can reduce the necessary data volume or model size to achieve a target performance, and that models are robust to moderate data corruption, with an effective data size decaying sublinearly with quality.

Key insights

• The quality-aware scaling law accurately models pretraining loss as a function of model size, data volume, and data quality, reducing to standard scaling laws when Q=1.
• Improvements in data quality quantitatively compensate for reductions in dataset size, offering a concrete trade-off for resource allocation and compute efficiency.
• LLMs exhibit robustness to moderate data corruption, with the effective dataset size decaying sublinearly with decreasing data quality (γ < 1), suggesting models retain useful information from partially corrupted samples.

Summary by alphaXiv.