Reproduce “Why mask diffusion does not work” on OpenResearch

About this paper

Haocheng Sun, Cynthia Xin Wen, Edward Hong Wang

Researchers from WhaleTech.ai demonstrate that mask diffusion language models inherently output conditional marginal distributions for individual masked tokens, not coherent joint probabilities, which limits true parallel generation and often causes distant predictions to homogenize. The study finds that effective generation with these models typically defaults to an autoregressive-like process, challenging the common assumption of their native parallel and bidirectional capabilities.

Key insights

• Mask diffusion models primarily learn to predict conditional marginal distributions for individual tokens, not joint probability distributions for multiple simultaneously masked tokens.
• The marginal probability distributions for masked tokens exhibit strong distance dependence: sharper near unmasked context and smoother, less informative for distant tokens.
• Distant masked token predictions tend to homogenize, often collapsing to high-frequency function words or end-of-text tokens, limiting the informativeness of parallel predictions.

Summary by alphaXiv.