Reproduce “Fixing a Broken ELBO” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, Kevin Murphy

This paper introduces an information-theoretic framework to analyze deep latent-variable models, specifically demonstrating that maximizing the Evidence Lower Bound (ELBO) can lead to latent variables being ignored by powerful decoders. It shows that adjusting the beta parameter in beta-VAEs offers a simple, theoretically grounded method to ensure latent code utilization, urging independent reporting of rate and distortion metrics for a more complete evaluation.

Key insights

  • Maximizing the Evidence Lower Bound (ELBO) does not inherently guarantee a useful latent representation, particularly with powerful decoders, as models can achieve high ELBO by ignoring the latent variable.
  • The Rate-Distortion (RD) curve offers a more informative diagnostic tool than ELBO, clearly illustrating the fundamental trade-off between compression (Rate) and reconstruction accuracy (Distortion) in latent variable models.
  • The beta-VAE objective is a principled method, viewed as a Legendre transform of the RD curve, for navigating this trade-off and controlling the amount of information encoded in the latent space.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.