Reproduce “WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiantao Qiu, Haijun Lv, Zhenjiang Jin, Rui Wang, Wenchang Ning, Jia Yu, ChaoBin Zhang, Zhenxiang Li and 18 more

Researchers at the Shanghai Artificial Intelligence Laboratory developed WanJuan-CC, an open-source English webtext dataset created from Common Crawl data through extensive filtering for safety and quality. Models trained on WanJuan-CC demonstrated reduced toxicity and superior performance on various downstream NLP tasks compared to models trained on RefinedWeb.

Key insights

  • A systematic, multi-stage filtering approach, including extensive deduplication and model-based classification, is critical for transforming raw web scrapes into high-utility, safe datasets for LLM training.
  • Prioritizing the retention of recent data during deduplication significantly enhances the overall quality and relevance of the resulting dataset.
  • Integrating PII masking and specialized classifiers for various safety attributes (toxicity, pornography) leads to demonstrably safer datasets compared to existing alternatives.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.