Reproduce “SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nan He, Weichen Xiong, Hanwen Liu, Yi Liao, Lei Ding, Kai Zhang, Guohua Tang, Xiao Han and 1 more
SoftDedup introduces a data reweighting approach that assigns sampling weights based on a quantified "data commonness" to accelerate Language Model pre-training. This method reduces required training steps by over 26% and improves downstream task accuracy by 1.77%, even on pre-deduplicated datasets, while avoiding information loss.
Key insights
- Soft deduplication, through its reweighting mechanism, offers a more effective and nuanced approach to handling data redundancy than traditional binary hard deduplication, improving model training and generalization.
- The method serves as a powerful complement to existing hard deduplication techniques, providing significant benefits even on datasets that have already undergone rigorous cleaning.
- Finer data partitioning (more segments) enhances downstream task performance, while larger weight disparities (more aggressive down-weighting of common data) consistently reduce model perplexity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.