Reproduce “SemDeDup: Data-efficient learning at web-scale through semantic deduplication” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, Ari S. Morcos
SemDeDup from Meta AI introduces a scalable method for identifying and removing semantic duplicates from web-scale datasets by leveraging embeddings from pre-trained foundation models. This approach significantly improves training efficiency and computational resource usage, allowing models to achieve comparable or better performance on tasks such as zero-shot image classification and language modeling with substantially reduced training data and time.
Key insights
- Web-scale datasets like LAION contain extreme semantic redundancy, with up to 50% of data being semantically similar despite perceptual differences.
- Leveraging embeddings from pre-trained foundation models allows for effective measurement of semantic similarity, which is crucial for identifying these duplicates.
- A scalable clustering-based approach can reduce the computational complexity of semantic deduplication from O(N^2) to O(N^2/k), making it feasible for massive datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.