Reproduce “Adaptive Momentum and Nonlinear Damping for Neural Network Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aikaterini Karoni, Rajit Rajpal, Benedict Leimkuhler, Gabriel Stoltz
The authors introduce two momentum methods for neural-network training: iKFAD, which adapts friction separately for each parameter based on its recent motion, and cubic damping (CD), which increases damping with instantaneous motion. Both avoid Adam-style per-parameter learning rates. Across tested transformer tasks, they narrow the loss gap between standard momentum SGD and Adam, though results vary by task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.