Reproduce “CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, Thien Huu Nguyen

CulturaX provides a meticulously curated, open-source multilingual dataset of 6.3 trillion tokens across 167 languages, addressing the scarcity and quality issues of existing LLM training data. This work, a collaboration between the University of Oregon and Adobe Research, demonstrates a scalable methodology for creating high-quality, web-sourced corpora.

Key insights

  • Automated, statistical metric-based cleaning using Interquartile Range (IQR) variants proves effective and scalable for curating massive multilingual datasets.
  • Rigorous multi-stage cleaning and comprehensive deduplication of web-scraped data can produce high-quality, large-scale training resources comparable to proprietary datasets.
  • Developing LLM-scale multilingual datasets openly is feasible with substantial computational resources and meticulous methodology, fostering greater research transparency and accessibility.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.