Reproduce “No MCMC for me: Amortized sampling for fast and stable training of energy-based models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Will Grathwohl, Jacob Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, David Duvenaud
VERA replaces training-time MCMC sampling for energy-based models with a learned generator and variational entropy estimation. On MNIST NICE models, it reports likelihood −1138 with entropy weight λ=1, versus −1219 for MEG; maximum-likelihood training reports −791. For CIFAR-10 JEM, authors report 2.8× faster training than the official implementation with default hyperparameters. Its objective is an upper bound on log likelihood.
Key insights
- The variational formulation provides a sufficiently tight bound on the EBM likelihood, allowing for high-quality density models without relying on exact MCMC sampling.
- Amortizing the sampling process with a trained generator significantly improves training speed and stability for EBMs, making them more practical to implement and converge.
- Entropy regularization is critical for VERA to achieve high-quality density models and diverse sample generation, preventing mode collapse and ensuring a more accurate representation of the data distribution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.