Reproduce “Practical Lossless Compression with Latent Variables using Bits Back Coding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

James Townsend, Tom Bird, David Barber

Researchers at UCL developed Bits Back with Asymmetric Numeral Systems (BB-ANS), a practical scheme for lossless compression with deep latent variable models, achieving near-optimal rates and outperforming generic algorithms like bz2 on MNIST (e.g., 0.19 bits/dimension for binarized MNIST compared to 0.25). This method effectively resolves long-standing implementation challenges for bits-back coding, bridging information theory with modern deep generative models.

Key insights

  • The LIFO nature of Asymmetric Numeral Systems (ANS) is uniquely suited for the chaining requirement of bits-back coding, resolving the practical issues that plagued previous arithmetic coding implementations.
  • The theoretical message length for bits-back coding with continuous latent variables is robust to the choice of discretization width (δy), as these terms cancel out in the expected rate calculation.
  • Optimizing a latent variable model by maximizing its Evidence Lower Bound (ELBO) directly corresponds to minimizing the expected message length in bits-back coding, providing a clear objective for training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.