Reproduce “BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training” on OpenResearch

About this paper

Nikitas Theodoropoulos, Giorgos Filandrianos, Vassilis Lyberatos, Maria Lymperaiou, Giorgos Stamou

Researchers at the Artificial Intelligence and Learning Systems Laboratory at the National Technical University of Athens explored using synthetic story data to augment language model pre-training within the BabyLM Challenge's data constraints. They demonstrated that GPT-Neo models can generate high-quality stories even when trained on limited TinyStories data and found that a balanced data sampling strategy yielded modest performance improvements for LTG-BERT on linguistic and general understanding benchmarks, especially in the 10M word budget setting.

Key insights

• Small generative models (GPT-Neo 41M) can acquire impressive grammatical and generative abilities with surprisingly little high-quality synthetic data, with a notable performance cutoff around 50M words.
• The method of combining synthetic and real data during training is crucial; simply adding more data can be detrimental, while balanced sampling strategies can lead to performance improvements.
• Generating more diverse text (e.g., via nucleus sampling) can benefit syntactic learning (BLiMP) but may introduce semantic inconsistencies that hinder knowledge-based tasks (GLUE/EWoK).

Summary by alphaXiv.