Reproduce “CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, Edouard Grave
CCNet provides an automated, efficient pipeline for extracting massive, high-quality monolingual text datasets for over 100 languages from raw Common Crawl data. This pipeline significantly improved pre-trained language model performance, particularly for low-resource languages, with CCNet-trained models outperforming Wikipedia-trained counterparts by an average of 3.3% on XNLI.
Key insights
- Performing paragraph-level deduplication *before* language identification significantly increases the number of identified documents, particularly for low-resource languages, by effectively removing English boilerplate.
- Perplexity, as calculated by Kneser-Ney LMs trained on high-quality domain data like Wikipedia, serves as a reliable and effective proxy for assessing text quality.
- The pipeline's design for massive parallelization (e.g., 5000 CPU cores) enables efficient processing of petabytes of raw web data in a matter of hours.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.