Reproduce “A Kernel Approach for Semi-implicit Variational Inference” on OpenResearch

About this paper

Longlin Yu, Ziheng Cheng, Shiyue Zhang, Cheng Zhang

Kernel Semi-implicit Variational Inference (KSIVI) reformulates the optimization of semi-implicit variational distributions using kernel methods, eliminating the auxiliary lower-level optimization problem found in prior approaches. It yields more stable and efficient posterior approximation, demonstrated by improved pairwise correlation capture in Bayesian logistic regression and robust handling of multimodal targets.

Key insights

• By constraining the auxiliary function in SIVI-SM to an RKHS, the intractable inner-loop optimization problem can be solved in closed-form, significantly simplifying the objective function.
• The marginal score of semi-implicit distributions, typically intractable, can be effectively replaced by its tractable conditional counterpart within the KSD objective, enabling practical computation via Monte Carlo.
• Appropriate kernel selection is crucial for accurate approximation, especially for non-Gaussian features like heavy tails, and hierarchical constructions can effectively mitigate issues like mode collapse.

Summary by alphaXiv.