Reproduce “Understanding Stochastic Natural Gradient Variational Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaiwen Wu, Jacob R. Gardner
Researchers from the University of Pennsylvania established the first O(1/T) non-asymptotic convergence rate for stochastic Natural Gradient Variational Inference (NGVI) applied to models with conjugate likelihoods. The study also revealed that the Evidence Lower Bound (ELBO) objective, when parameterized by expectation parameters for non-conjugate likelihoods, is inherently non-convex, providing new insights into NGVI's optimization landscape.
Key insights
- Stochastic NGVI achieves an O(1/T) non-asymptotic convergence rate for conjugate likelihoods, matching the asymptotic rate of SGD but demonstrating superior constant factors that are independent of the objective's condition number or initialization.
- For non-conjugate likelihoods, the ELBO objective, when canonically optimized by stochastic NGVI in the expectation parameter space, is generally non-convex, presenting a more complex optimization landscape than previously assumed in some contexts.
- Despite non-convexity for non-conjugate models, the negative ELBO still possesses a unique minimizer and stationary point for log-concave likelihoods, indicating that a global optimum is reachable even if the optimization path is more involved.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.