Reproduce “Avoiding Latent Variable Collapse With Generative Skip Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Adji B. Dieng, Yoon Kim, Alexander M. Rush, David M. Blei

Generative skip models are introduced for Variational Autoencoders (VAEs) to mitigate latent variable collapse, an issue where VAEs fail to learn meaningful data representations. This architectural modification adds direct connections from the latent variable to multiple layers of the decoder, demonstrably increasing mutual information between observations and latent variables and leading to more effective utilization of the latent space across image and text data.

Key insights

  • Generative skip models inherently promote higher mutual information between the observations `x` and the latent variables `z` (`I_p_skip-vae(x, z) ≥ I_p_vae(x, z)`), as proven by the authors.
  • By directly connecting `z` to multiple layers, the method ensures that the latent variable maintains a stronger and more persistent influence on the generated output, preventing information about `z` from diminishing through deeper layers.
  • This mechanism implicitly biases the generative model to depend more on `z`, thereby forcing the inference network to learn more data-dependent posterior distributions and preventing posterior collapse.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.