Reproduce “Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform” on OpenResearch

About this paper

Jun Li, Li Fuxin, Sinisa Todorovic

Researchers at Oregon State University developed efficient Riemannian optimization algorithms, Cayley SGD and Cayley ADAM, that strictly enforce orthonormality constraints on deep neural network parameters by utilizing iterative approximations for retraction maps and implicit vector transport. These methods achieve faster convergence and comparable or improved performance while significantly reducing the per-epoch training time compared to other hard orthonormality methods.

Key insights

• The closed-form Cayley transform can be efficiently approximated through a rapid iterative process using only matrix multiplications, making it practical for large-scale deep learning applications on the Stiefel manifold.
• Momentum can be implicitly incorporated into Riemannian optimization on the Stiefel manifold by combining gradients and previous momentum in Euclidean space, avoiding computationally expensive explicit vector transport operations.
• Strict orthonormality constraints, when enforced efficiently, lead to faster convergence, improved training stability, and comparable or superior model performance in deep neural networks across various architectures.

Summary by alphaXiv.