Reproduce “AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li and 4 more
AR-DIFFUSION introduces an auto-regressive diffusion model for text generation that integrates sequential dependency into the denoising process. This model achieves quality comparable to auto-regressive Transformers while significantly accelerating inference speed, making diffusion models more practical for various text generation tasks.
Key insights
- Integrating auto-regressive-like sequential processing into diffusion models can bridge the quality gap between AR and NAR text generation.
- By varying denoising speeds across tokens based on their position, diffusion models can effectively capture the left-to-right dependencies of language.
- Significant inference speedups are achievable in text diffusion models by carefully designed skipping mechanisms that reduce denoising steps without substantial quality degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.