Reproduce “GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Abir De, Rishabh Iyer
The GRAD-MATCH framework introduces a method for data subset selection that enables efficient deep model training by directly minimizing a theoretically derived gradient matching error. It achieves substantial training speedups (e.g., up to 7x on CIFAR-10, 3x on ImageNet) while maintaining competitive generalization performance and demonstrating robustness in imbalanced datasets.
Key insights
- The convergence of adaptive data subset selection algorithms is critically dependent on minimizing the gradient mismatch between the selected subset and the full dataset.
- The gradient matching objective function exhibits a property of weak submodularity, enabling the use of efficient greedy algorithms like OMP with approximation guarantees.
- Combining theoretical gradient matching with practical approximations (e.g., last-layer, per-batch, warm-starting) allows for efficient and scalable data selection for deep learning, even on very large datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.