Reproduce “Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chen-Hao Chao, Wei-Fang Sun, Hanwen Liang, Chun-Yi Lee, Rahul G. Krishnan
This work introduces MDM-Prime, a discrete diffusion model utilizing a partial masking scheme that represents tokens as sub-token sequences to enable fine-grained denoising. This approach significantly reduces redundant 'idle steps' during sampling, leading to improved computational efficiency and achieving superior perplexity in text generation compared to autoregressive models, alongside competitive performance in image synthesis.
Key insights
- Decomposing discrete tokens into sub-tokens for partial masking drastically reduces 'idle steps' and improves model utilization during reverse diffusion by making each step more informative.
- Modeling the joint probability of sub-tokens is crucial for generating valid and high-quality tokens, effectively addressing the challenge of preventing invalid sub-token sequences.
- Order-agnostic discrete diffusion models can achieve state-of-the-art performance in text generation perplexity, surpassing autoregressive models, by refining the granularity of the denoising process without introducing sequential dependencies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.