Reproduce “Unifying Masked Diffusion Models with Various Generation Orders and Beyond” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chunsan Hong, Sanghyun Lee, Jong Chul Ye
Researchers from KAIST introduce a unified theoretical framework, Order-expressive Masked Diffusion Model (OeMDM), for understanding various generation orders in masked diffusion models. Building on this, they develop Learnable-order Masked Diffusion Model (LoMDM), which jointly learns context-dependent generation orders and the diffusion backbone from scratch, achieving up to 3 points lower test perplexity on LM1B and OpenWebText while being approximately 3 times faster to train than standard MDLM.
Key insights
- A generalized NELBO, which includes an order mismatch loss (L_velocity), can theoretically unify various MDM paradigms and autoregressive models, demonstrating that explicit modeling of generation order is crucial for optimal performance.
- Jointly learning context-dependent generation orders with the diffusion backbone through a single objective significantly improves model performance and training efficiency compared to two-stage approaches or fixed orders.
- The learned generation policy in LoMDM adopts a 'coarse-to-fine' strategy, prioritizing structural words early in the decoding process and semantic content later, indicating a more effective approach for unconditional text generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.