Reproduce “Improving Lossless Compression Rates via Monte Carlo Bits-Back Coding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yangjun Ruan, Karen Ullrich, Daniel Severo, James Townsend, Ashish Khisti, Arnaud Doucet, Alireza Makhzani, Chris J. Maddison
McBits uses multiple Monte Carlo samples in a bits-back coding construction to tighten the rate bound without requiring a more accurate approximate posterior. It represents importance-sampling estimates over an expanded latent space; under consistency and convergence conditions, the net rate approaches the negative log marginal likelihood as sample count grows. Experiments report gains on EMNIST transfer and sequential music.
Key insights
- Extending the latent space to incorporate Monte Carlo samples and reinterpreting tighter variational bounds allows for the asymptotic elimination of the KL divergence gap in bits-back coding.
- Leveraging common uniform random numbers through coupling techniques effectively controls the initial bit overhead of Monte Carlo-based bits-back schemes, improving their practical efficiency without compromising compression rate gains.
- Specialized Monte Carlo methods, such as Sequential Monte Carlo, are more effective for compressing sequential data, exhibiting faster convergence to optimal rates compared to general importance sampling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.