Reproduce “Blu-WERP (Web Extraction and Refinement Pipeline): A Scalable Pipeline for Preprocessing Large Language Model Datasets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gowtham, Sai Rupesh, Sanjay Kumar, Saravanan, Venkata Chaithanya
Blu-WERP (Web Extraction and Refinement Pipeline) curates high-quality web-scale datasets for large language model training by systematically integrating advanced extraction, multi-level deduplication, and benchmark-targeted semantic filtering. This pipeline leads to a 4.0% aggregate accuracy improvement over DCLM and a 9.5% improvement over FineWeb for 1B parameter models, showcasing enhanced training efficiency and model performance.
Key insights
- High-quality data curation, through systematic preprocessing, is a primary driver of LLM performance, surpassing architectural innovations in importance.
- Lightweight yet robust components, like JusText for text extraction and Bloom Filter for deduplication, can achieve superior data quality and processing efficiency when optimally integrated.
- Benchmark-driven semantic filtering (BETR) effectively aligns the data curation process with the actual performance metrics of downstream language models.
- Rigorous, component-level ablation studies are crucial for identifying the most effective strategies within a complex data pipeline, leading to quantifiable performance gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.