Reproduce “Simple Denoising Diffusion Language Models” on OpenResearch

About this paper

Huaisheng Zhu, Zhengyu Chen, Shijie Zhou, Zhihui Xie, Yige Yuan, Shiqi Chen, Zhimeng Guo, Siyuan Xu and 3 more

Zhu et al. developed Simple Denoising Diffusion Language Models (SDDLM), a framework that simplifies the training objective for Uniform State Diffusion Models (USDMs) to reduce computational overhead. An enhanced variant, SDDLM-V1, incorporates an Anti-Uniform Distribution Sharpening regularization, leading to improved generative sample quality, achieving a Gen PPL of 45.18 on OWT compared to Duo's 80.43, while maintaining strong scalability.

Key insights

• Focusing the denoising loss solely on perturbed tokens is crucial for stable optimization and efficient training in discrete diffusion models, preventing training collapse.
• Explicitly penalizing overly uniform predictions through regularization significantly enhances the quality of generated samples in USDMs, particularly under high noise levels.
• A consistent discrepancy exists between ELBO-based perplexity and actual generative sample quality in discrete diffusion models, suggesting the former may not always accurately reflect generative capabilities.

Summary by alphaXiv.