Reproduce “Mixture weights optimisation for Alpha-Divergence Variational Inference” on OpenResearch

About this paper

Kamélia Daudel, Randal Douc

Researchers at Télécom Paris and Télécom SudParis present a complete global convergence proof for the Power Descent algorithm in alpha-divergence variational inference for mixture models when alpha is less than 1. They introduce the Renyi Descent algorithm, demonstrating its O(1/N) convergence rate, and show Power Descent's ability to cover forward Kullback-Leibler minimization as alpha approaches 1, enhancing the theoretical and practical robustness of alpha-divergence methods.

Key insights

• Global convergence for Power Descent in the mass-covering regime (alpha < 1) for mixture models can be rigorously proven by leveraging fixed-point properties and monotonic objective decrease, without assuming limit existence.
• Power Descent provides a unified framework for alpha-divergence minimization, as it smoothly transitions to an Entropic Mirror Descent algorithm for forward KL minimization when alpha approaches 1.
• The Renyi Descent algorithm, closely related to Power Descent, achieves a theoretically stronger O(1/N) convergence rate, offering a more efficient approach for alpha-divergence minimization than typical EMD algorithms.

Summary by alphaXiv.