Reproduce “A theoretical perspective on mode collapse in variational inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Roman Soletskyi, Marylou Gabrié, Bruno Loureiro

A theoretical study investigates mode collapse in variational inference, demonstrating it stems from the optimization objective rather than model expressivity. The work identifies mean alignment and vanishing weights as primary mechanisms, supported by a low-dimensional dynamical system analysis for Gaussian mixtures and validated with Normalizing Flows.

Key insights

  • Mode collapse in variational inference is primarily driven by two mechanisms: mean alignment (multiple variational components converging to a single target mode) and vanishing weights (components' weights reducing to zero).
  • A critical threshold `R_c` for target mode separation exists, beyond which mode-collapsed states become stable fixed points in the optimization landscape.
  • Theoretical insights derived from simplified Gaussian mixture models qualitatively and quantitatively align with observations from more complex Normalizing Flows, suggesting general applicability.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.