Reproduce “Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training” on OpenResearch

About this paper

Jiahui Peng, Xinlin Zhuang, Jiantao Qiu, Ren Ma, Jing Yu, He Zhu, Conghui He

A new data mixing strategy for large language model pre-training leverages semantic topics instead of data sources, yielding consistent performance improvements across various downstream benchmarks, including up to 1.90 points in reading comprehension tasks for 1.3B models and a wider performance gap for 3.3B models.

Key insights

• Organizing pre-training data by intrinsic semantic properties (topics) consistently outperforms organization by provenance (sources) across various LLM mixing algorithms and model scales.
• Topic-based data partitioning creates a more favorable optimization landscape during training, leading to better modeling of the relationship between mixture weights and model performance, as revealed by proxy model analysis.
• Combining both topic and source dimensions (e.g., via Cartesian product) or integrating data mixing with data selection techniques provides complementary information, further enhancing LLM performance.

Summary by alphaXiv.