Reproduce “OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, Ji Pei

Researchers at Tsinghua University and OpenCSG developed the OpenCSG Chinese Corpus, a series of openly accessible, high-quality Chinese datasets, to address the scarcity of robust training resources for Chinese Large Language Models. These datasets, comprising filtered web data, synthetic textbook-style content, and diverse conversational data, were shown to improve pretraining efficiency and significantly enhance instruction-following capabilities in 2B Llama-based models.

Key insights

  • Rigorous, LLM-driven quality scoring and filtering methods are essential for effectively transforming coarse Chinese web data into high-value educational content for LLM pretraining.
  • While synthetic data generation can scale knowledge-intensive content, careful prompt engineering and genre diversity are critical to avoid homogeneity and maximize its utility for robust LLM generalization.
  • Instruction fine-tuning with diverse, multi-turn conversational data covering a wide range of tasks significantly improves LLM alignment and their ability to follow complex instructions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.