Reproduce “Accelerating Deep Learning with Dynamic Data Pruning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ravi S Raju, Kyle Daruwalla, Mikko Lipasti
Researchers from the University of Wisconsin-Madison developed a dynamic data pruning framework that adaptively selects training samples throughout the deep learning process. This approach reduced total training time by up to 2x on CIFAR-10 while maintaining model accuracy, effectively addressing the high computational overhead and fixed nature of prior static data pruning methods.
Key insights
- Datasets consist of "always," "sometimes," and "never" samples, where "sometimes" samples exhibit transient importance during training and are missed by static methods.
- Dynamic data pruning, even with random sample selection, can outperform static baselines, demonstrating that continuous adaptation to sample importance is more effective than pre-selecting a fixed subset.
- The computational overhead of dynamic sample scoring is negligible compared to the time saved from reducing overall training iterations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.