Reproduce “Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shabnam Daghaghi, Benjamin Coleman, Benito Geordie, Anshumali Shrivastava
An adaptive sampling algorithm leverages an efficient Nadaraya-Watson Sketch (NWS) to predict data point importance, effectively accelerating deep neural network training. This approach achieved 1.3x to 1.9x faster convergence to baseline accuracy and often improved final model accuracy on various NLP tasks.
Key insights
- Nonparametric kernel regression can be made computationally efficient for large-scale applications through sketch-based approximations like NWS, offering O(1) inference time.
- Adaptive data sampling, when guided by an efficiently estimated importance score, can significantly accelerate deep learning training while preserving or improving accuracy.
- The dynamic interaction between a deep learning model's evolving state and an adaptive importance sampling distribution can lead to more efficient and effective learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.