Reproduce “The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text” on OpenResearch

About this paper

Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, Chloé Clavel

This work empirically demonstrates that recursively training large language models on synthetic text consistently reduces the linguistic diversity of their outputs across lexical, syntactic, and semantic dimensions. The decline is most pronounced in tasks requiring high creativity, highlighting a homogenization of AI-generated language.

Key insights

• Recursive training on synthetic data consistently leads to a substantial decline in linguistic diversity across lexical and syntactic dimensions in LLM outputs.
• Tasks requiring higher creativity (high-entropy tasks) exhibit a more pronounced and rapid reduction in linguistic diversity.
• Syntactic diversity, quantified by a novel graph-based metric, shows a significant decline comparable to lexical diversity, highlighting its importance in language richness.
• Small interventions, such as filtering low-quality synthetic data or mixing a minor percentage of fresh human data, provide only minimal mitigation to the observed diversity decline.

Summary by alphaXiv.