Reproduce “MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Da Chang, Ganzhao Yuan
MGUP introduces a momentum-gradient alignment update policy that selectively scales learning rates for parameters based on their alignment, providing fine-grained intra-layer control for stochastic optimizers. It offers theoretical convergence guarantees and empirically demonstrates faster convergence, improved generalization, and enhanced stability across large language model pretraining and downstream fine-tuning tasks compared to standard and cautious baselines.
Key insights
- Theoretical analysis demonstrates that a strictly positive decay factor ($\gamma > 0$) for less-aligned parameters is crucial for stable convergence in stochastic settings, preventing issues like those found in zero-update policies.
- MGUP-AdamW achieves an expected convergence rate of $O(\log(T)/\sqrt{T})$ and a high-probability convergence rate of $O(\text{poly}(\log(T))/\sqrt{T})$ for non-convex optimization.
- The average effective learning rate for MGUP is inherently higher than the base optimizer, contributing to observed acceleration while maintaining stability through differentiated updates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.