Reproduce “Optimization Guarantees for Square-Root Natural-Gradient Variational Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Navish Kumar, Thomas Möllenhoff, Mohammad Emtiyaz Khan, Aurelien Lucchi
This research provides rigorous exponential convergence guarantees for Square-Root Natural-Gradient Variational Inference (NGVI) when applied to strongly concave log-likelihoods, enabled by a novel square-root parameterization for Gaussian covariance. The discrete-time algorithm, Square-Root Variational Newton (SR-VN), demonstrates faster convergence empirically compared to methods based on Euclidean or Wasserstein geometries.
Key insights
- Parameterizing the Gaussian covariance using its Cholesky factor preserves the convexity of the expected log-likelihood, enabling the application of convex optimization theory for NGD analysis.
- A Riemannian Polyak-Łojasiewicz (PL) inequality holds for the KL functional in the square-root parameterization, which is crucial for establishing exponential convergence rates.
- Natural gradient methods, including SR-VN, inherently exhibit faster convergence properties due to their preconditioned updates, outperforming methods based on Euclidean or Wasserstein geometries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.