Reproduce “Small-to-Large Generalization: Data Influences Models Consistently Across Scale” on OpenResearch

About this paper

Alaa Khaddaj, Logan Engstrom, Aleksander Madry

Researchers at MIT empirically demonstrate that insights into training data distribution effects, gained from smaller language models, largely translate to their larger counterparts, showing R² correlations often exceeding 0.9 across various benchmarks. This validates the use of proxy models for data-centric machine learning tasks, offering significant computational savings.

Key insights

• Training data effects exhibit strong consistency across different model scales, with correlations (R²) frequently above 0.9.
• The degree of correlation can vary by task, with some benchmarks like SQuAD showing more modest R² values (0.46-0.65) compared to others (e.g., HellaSwag at 0.99).
• Proxy models can reliably predict the relative impact of data changes on larger models, even when the proxy model itself demonstrates low accuracy on the given task.

Summary by alphaXiv.