Reproduce “SelMatch: Effectively Scaling Up Dataset Distillation via Selection-Based Initialization and Partial Updates by Trajectory Matching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yongmin Lee, Hye Won Chung
SelMatch, developed by researchers at KAIST, introduces a method for scaling dataset distillation by integrating selection-based initialization and partial updates through trajectory matching. This approach consistently outperforms existing methods across various benchmarks and architectures, achieving a 3.5% accuracy increase on CIFAR-100 (IPC=50) compared to the next best technique.
Key insights
- An optimal synthetic dataset for larger IPCs benefits from initializing with and preserving a greater proportion of difficult and diverse samples from the original dataset.
- Freezing a portion of pre-selected, complex samples during distillation effectively maintains diversity and ensures coverage of hard-to-learn features.
- Intelligently selecting an initial subset of data can alone provide a strong baseline, demonstrating the importance of data composition beyond just distillation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.