Reproduce “Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts” on OpenResearch

About this paper

Jiayuan Ye, Vitaly Feldman, Kunal Talwar

Training data pruning, guided by loss-based selection, improves factual memorization in language models. This method enabled a 110M parameter model to memorize 1.3 times more entity facts from Wikipedia, matching the performance of a 1.3B parameter model trained conventionally, without degrading general language understanding abilities.

Key insights

• The observed limited factual accuracy in LLMs is often a consequence of training data distributions exceeding model capacity, especially with skewed fact frequencies.
• Training loss provides a practical and effective proxy for fact frequency and entropy, enabling principled data selection without explicit fact annotations.
• Intelligently pruning training data based on loss allows smaller language models to achieve factual memorization capabilities comparable to significantly larger models.

Summary by alphaXiv.