Reproduce “Gradient-free variational learning with conditional mixture networks” on OpenResearch

About this paper

Conor Heins, Hao Wu, Dimitrije Markovic, Alexander Tschantz, Jeff Beck, Christopher Buckley

CAVI-CMN makes approximate Bayesian learning practical for two-layer conditional mixture networks through analytic coordinate-ascent updates, using linear experts selected by a probabilistic gate and Pólya–Gamma augmentation to retain distributions over parameters and latent variables. Across classification benchmarks, it matches or sometimes exceeds MLE in accuracy and improves probabilistic metrics over MLE. On Breast Cancer, its WAIC is −0.0504 versus −0.3133 for MLE, but it does not consistently outperform Bayesian alternatives and runs more slowly as latent dimension grows.

Key insights

• Careful design of network architecture (Conditional Mixture Networks) combined with conditionally conjugate priors allows for closed-form, gradient-free variational inference updates.
• Pólya-Gamma augmentation is instrumental in transforming non-conjugate logistic regression components into a form amenable to analytic updates, enabling efficient Bayesian inference.
• Efficient, gradient-free Bayesian learning can match the probabilistic performance of computationally intensive sampling methods while offering significantly faster convergence and lower runtime.

Summary by alphaXiv.