Reproduce “Pathfinder: Parallel quasi-Newton variational inference” on OpenResearch

About this paper

Lu Zhang, Bob Carpenter, Andrew Gelman, Aki Vehtari

Researchers at Columbia University, Flatiron Institute, and Aalto University developed Pathfinder, a variational inference method that generates approximate posterior draws by leveraging quasi-Newton optimization trajectories and local normal approximations. It demonstrates one to two orders of magnitude fewer log-density and gradient evaluations compared to ADVI and short HMC runs while maintaining comparable or superior approximation quality, and significantly improves MCMC initialization.

Key insights

• Leveraging curvature information from quasi-Newton optimization provides a stable and efficient way to construct local normal approximations of the posterior.
• Parallelization of optimization paths and ELBO evaluations significantly accelerates the inference process compared to intrinsically sequential VI methods.
• Combining multiple local approximations with robust importance resampling (PS-IR) enhances the method's ability to handle multimodality and improve approximation quality.

Summary by alphaXiv.