Reproduce “CCI4.0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guang Liu, Liangdong Wang, Jijie Li, Yang Yu, Yao Xu, Jiabei Chen, Yu Bai, Feng Liao and 1 more
CCI4.0 is a 35TB bilingual (Chinese-English) pretraining dataset that integrates 4.5 billion synthesized Chain-of-Thought (CoT) templates to improve reasoning capabilities in large language models. Models trained on CCI4.0 achieve comparable performance to those trained on 30 billion tokens of Nemotron-CC-HQ with only 10 billion tokens, and demonstrate enhanced performance on Chinese benchmarks and reasoning tasks.
Key insights
- Explicitly integrating Chain-of-Thought patterns during pretraining can accelerate reasoning development and reduce hallucination in LLMs.
- A meticulously designed multi-stage pipeline, including advanced deduplication and multi-dimensional quality control, is crucial for building high-quality, large-scale datasets.
- Bilingual datasets with specialized curation for each language can achieve balanced and strong performance across both languages, addressing gaps in existing resources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.