Reproduce “KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Truong Thao Nguyen, Balazs Gerofi, Edgar Josafat Martinez-Noriega, François Trahay, Mohamed Wahib
KAKURENBO presents an adaptive, online deep learning framework that reduces neural network training time by dynamically hiding less important samples from the backward pass. The method achieves up to a 22.4% reduction in total training time while maintaining or slightly improving prediction accuracy on large-scale datasets such as ImageNet-1K and DeepCAM.
Key insights
- The importance of individual training samples changes dynamically throughout the learning process, supporting an adaptive, epoch-by-epoch hiding strategy over static pruning.
- Adjusting the learning rate proportionally to the fraction of hidden samples is crucial for maintaining convergence speed and final model accuracy when reducing the number of SGD steps.
- Filtering hidden samples based on both low loss and high historical prediction confidence prevents mistakenly omitting critical or challenging data points from training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.