Reproduce “Monte Carlo Variational Auto-Encoders” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Achille Thin, Nikita Kotelevskii, Arnaud Doucet, Alain Durmus, Eric Moulines, Maxim Panov

The authors introduce Monte Carlo variational autoencoders (MCVAEs), which train VAEs with Markov-chain approximations of latent posteriors. They use annealed or sequential importance sampling to tighten the evidence lower bound and derive gradient estimators, avoiding a learned reverse transition. On dynamically binarized MNIST, Langevin SIS reports an NLL estimate of 84.58 ± 0.39 versus 84.82 ± 0.10 for IWAE; added transitions mean this is not compute-matched.

Key insights

  • Advanced Monte Carlo methods (SIS and AIS) can be effectively integrated into VAE training by devising strategies for unbiased gradient estimation for MCMC steps and simplifying the specification of backward kernels, for instance, by approximating them with forward kernels.
  • L-MCVAE, by approximating backward kernels, offers more stable training due to lower gradient variance compared to A-MCVAE, even though A-MCVAE can achieve tighter ELBOs, highlighting a trade-off that can be managed with variance reduction techniques.
  • The use of Langevin dynamics and MCMC transitions facilitates the optimization process, leading to faster convergence and lower held-out log-likelihoods during training compared to standard VAEs and IWAEs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.