Reproduce “HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Feihu Huang, Guanyi Zhang, Songcan Chen
HomeAdam and HomeAdamW are novel adaptive gradient optimization algorithms that achieve an improved theoretical generalization error of O(1/N), matching SGD's performance. These algorithms incorporate a conditional switching mechanism to revert to momentum-based SGD-like updates when adaptive learning rates risk poor generalization, demonstrating superior empirical performance across computer vision and natural language processing tasks.
Key insights
- Standard Adam and AdamW variants, especially square-root-free ones, can exhibit significantly worse theoretical generalization (O(hat{rho}^-2T)/N) and slower convergence (O(check{rho}^-1 T^-1/4)) due to very small adaptive scaling factors.
- A principled 'return home' mechanism, switching to an SGD-like update, can prevent problematic adaptive scaling, thereby leading to improved generalization without compromising convergence speed.
- Decoupled weight decay (HomeAdamW) consistently enhances generalization within the HomeAdam framework, aligning with its known benefits in other adaptive optimizers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.