Reproduce “LP Data Pipeline: Lightweight, Purpose-driven Data Pipeline for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yungi Kim, Hyunsoo Ha, Seonghoon Yang, Sukyung Lee, Jihoo Kim, Chanjun Park
Upstage AI's LP Data Pipeline introduces a fully CPU-based framework for curating high-quality, purpose-driven datasets for large language models. This efficient system processed a 4TB CommonCrawl dump in under 4.5 hours for approximately $350, generating substantial domain-specific corpora for both English and Korean.
Key insights
- High-quality LLM data curation is feasible and efficient using only CPU-based resources, offering a cost-effective alternative to GPU-intensive methods.
- Strategic sequencing of data processing steps, from raw extraction to fine-grained filtering, significantly enhances efficiency and scalability.
- Leveraging signals from KenLM models trained on both high-quality and low-quality data effectively enables a CPU-based FastText model to perform robust quality classification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.