Reproduce “Regurgitative Training: The Value of Real Data in Training Large Language Models” on OpenResearch

About this paper

Jinghui Zhang, Dandan Qiao, Mochen Yang, Qiang Wei

Training Large Language Models (LLMs) on content generated by other LLMs, termed "regurgitative training," generally degrades or stalls performance compared to using real human-generated data. This degradation is attributed to higher error rates and lower lexical diversity in synthetic data, though mitigation strategies like prioritizing quality or AI-detection based filtering can partially reduce performance loss.

Key insights

• Regurgitative training consistently leads to performance degradation or stalling, emphasizing the unique and irreplaceable value of real human-generated data for LLM training.
• LLM-generated data exhibit higher error rates and lower lexical diversity, which are key mechanisms driving the observed performance losses and reduced generalization abilities.
• The impact of regurgitative training is dependent on the capability of the LLM generating the synthetic data; more capable generators produce less harmful, and sometimes beneficial, data.

Summary by alphaXiv.