Reproduce “Infinite Mask Diffusion for Few-Step Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jaehoon Yoo, Wonjung Kim, Chanhyuk Lee, Seunghoon Hong
Researchers from KAIST developed the Infinite Mask Diffusion Model (IMDM), which replaces the deterministic single-state mask in standard Masked Diffusion Models (MDMs) with a stochastic infinite-state mask. This design theoretically mitigates a fundamental factorization error, allowing MDMs to achieve high-quality few-step text generation, exemplified by a nearly 50% reduction in generative perplexity in 2-step generation on LM1B and matching 64-step teacher performance with only 4 steps in conditional generation on OpenWebText.
Key insights
- Standard MDMs possess a non-trivial theoretical lower bound on factorization error due to the deterministic nature of their single-state mask, regardless of the distillation method used.
- IMDM's stochastic infinite-state mask theoretically enables zero factorization error by providing sufficient randomness through a 'partition-and-map' mechanism to recover true conditional probabilities.
- The IMDM design allows structural compatibility with pre-trained MDMs, enabling direct fine-tuning and integration into existing few-step distillation frameworks without requiring full re-training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.