Reproduce “Latent Diffusion for Language Generation” on OpenResearch

About this paper

Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, Kilian Q. Weinberger

Cornell University researchers introduce Latent Diffusion for Language Generation (LD4LG), a two-stage framework that successfully adapts continuous diffusion models to discrete natural language by operating in the compact latent space of a pre-trained encoder-decoder language model. This approach dramatically outperforms previous diffusion-based language models (e.g., MAUVE 0.716 vs 0.043 on ROCStories) and achieves performance competitive with autoregressive baselines across various generation tasks, while also showing less susceptibility to memorization.

Key insights

• Diffusion models and pre-trained language models are complementary; leveraging existing PLMs' robust linguistic knowledge and representational power through their latent spaces is crucial for effective text diffusion.
• Operating diffusion models in a compact, fixed-length, continuous latent space derived from a high-quality language autoencoder is more effective and efficient for discrete data than direct discrete diffusion or continuous diffusion in raw embedding spaces.
• Architectural choices like self-conditioning and v-parameterization, successfully applied in image diffusion, significantly improve the quality and efficiency of latent text diffusion.
• The proposed framework provides a versatile approach for various language generation tasks, demonstrating that the challenges of applying diffusion to text can be overcome by intelligently combining existing strengths in generative AI.

Summary by alphaXiv.