Reproduce “Adaptive Learning Rate via Covariance Matrix Based Preconditioning for Deep Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yasutoshi Ida, Yasuhiro Fujiwara, Sotetsu Iwamura
Researchers at NTT Software Innovation Center developed SDProp, an adaptive learning rate algorithm that uses covariance matrix-based preconditioning to explicitly model and mitigate gradient noise in deep neural network training. It achieved faster convergence and improved training accuracy across various architectures, particularly with smaller mini-batch sizes, compared to RMSProp and Adam.
Key insights
- Explicitly modeling gradient noise via a covariance matrix allows for more robust preconditioning than traditional Hessian approximations.
- Adapting learning rates based on the *centered variance* of gradients (oscillation) directly addresses noise, distinguishing it from methods relying on *uncentered variance* (magnitude).
- Exponential moving averages can effectively update the covariance matrix online, ensuring mathematical properties for stable optimization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.