Reproduce “Data Distillation: A Survey” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Noveen Sachdeva, Julian McAuley
A comprehensive survey establishes a formal framework and taxonomy for data distillation, a technique that synthesizes concise, high-fidelity data summaries to replace extensive datasets in deep learning applications. The research highlights how these methods reduce training time and computational resources while enabling models to achieve performance comparable to those trained on full datasets, especially when data is limited.
Key insights
- Data distillation presents a principled counterpoint to the "scale-is-everything" paradigm, focusing on optimizing data fidelity and efficiency rather than just increasing dataset size.
- Most data distillation methods solve a bilevel optimization problem, where an inner loop optimizes a learning algorithm on a data summary, and an outer loop optimizes the summary itself.
- Beyond accelerating model training, data distillation has potential applications in differential privacy, neural architecture search, continual learning, and federated learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.