Reproduce “Autoregressive Diffusion Models” on OpenResearch

About this paper

Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, Tim Salimans

Autoregressive Diffusion Models (ARDMs) unify order-agnostic autoregressive models and discrete diffusion models by introducing a flexible framework that supports parallel generation. The models achieve competitive likelihoods with significantly fewer generation steps than traditional discrete diffusion models and establish a new state of the art in lossless compression for individual images.

Key insights

• ARDMs theoretically unify order-agnostic autoregressive models and absorbing discrete diffusion models under a single framework, demonstrating their maximal expressiveness bounded by data dimensionality.
• The order-agnostic training objective and flexible parametrization allow for the use of standard deep learning architectures, eliminating the need for complex causal masking.
• Structuring the generative process through depth upscaling significantly enhances performance for high-cardinality discrete data by breaking down complex predictions into manageable stages.

Summary by alphaXiv.