Reproduce “Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang, Zhi Zheng, Jie Zhou and 4 more

Ultra-FineWeb is a high-quality pre-training dataset for Large Language Models, constructed using an efficient, cost-effective data filtering pipeline developed by ModelBest Inc., Tsinghua University, and Soochow University. This pipeline reduces data verification costs by over 90% and yields LLMs with significant performance gains across English and Chinese benchmarks.

Key insights

  • An efficient, "nearly-trained LLM" verification strategy provides reliable data quality feedback at a fraction of the cost of full training.
  • Classifier performance primarily benefits from initial, well-chosen diverse seed data; extensive iterative self-refinement of the classifier offers diminishing or negative returns.
  • Intersecting positive samples from multiple quality classifiers can yield even higher quality data and further improve LLM performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.