Reproduce “Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lingkai Kong, Yuqing Wang, Molei Tao
Researchers at Georgia Institute of Technology introduce the Momentum Stiefel Optimizer, a new gradient-based algorithm for orthogonality-constrained optimization. This optimizer maintains exact Stiefel manifold and tangent bundle structure while efficiently incorporating momentum, leading to improved performance in tasks like Projection Robust Wasserstein Distance and enhanced Vision Transformer accuracy on CIFAR-10 (8.32% error) and CIFAR-100 (30.20% error).
Key insights
- Deriving discrete-time optimizers from continuous-time variational dynamics can provide a principled framework for ensuring exact preservation of complex manifold and tangent bundle structures.
- Efficiently incorporating momentum in Riemannian optimization is achievable by geometrically decomposing momentum and employing operator splitting with carefully designed structure-preserving numerical integrators.
- Imposing orthogonality constraints, specifically within individual attention heads of Vision Transformers, can significantly improve model accuracy, indicating the practical benefits of precise geometric constraints with suitable optimizers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.