Reproduce “Improving Generalization and Convergence by Enhancing Implicit Regularization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingze Wang, Jinbo Wang, Haotian He, Zilin Wang, Guanhua Huang, Feiyu Xiong, Zhiyu Li, Weinan E and 1 more
The Implicit Regularization Enhancement (IRE) framework accelerates the implicit tendency of optimizers to find flatter, more generalizable solutions in deep neural networks. This framework improves generalization across various models and datasets while achieving up to 2.3x speed-ups in large language model pre-training at only 1.1x computational overhead per step.
Key insights
- The inherent slowness of implicit sharpness regularization can be addressed by a computationally efficient framework that targets specific dynamics.
- Selectively accelerating updates along flat directions, identified by approximated local curvature, is crucial for both faster convergence to flatter minima and improved generalization.
- A lightweight, diagonal Hessian estimation is sufficient for identifying flat directions, enabling significant improvements with minimal additional computational cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.