Reproduce “SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, Songfang Huang

SeqDiffuSeq enhances continuous diffusion models for sequence-to-sequence text generation by integrating an encoder-decoder Transformer architecture, self-conditioning, and an adaptive token-level noise schedule. The model demonstrates superior generation quality and 3.56x faster inference compared to its direct diffusion-based competitor, DiffuSeq, while also achieving competitive results against auto-regressive and non-auto-regressive baselines across five NLP tasks.

Key insights

  • An encoder-decoder Transformer architecture provides substantial computational benefits for conditional text diffusion, making inference significantly faster by avoiding repeated input encoding.
  • Self-conditioning and a token-level adaptive noise schedule are complementary and essential for enhancing the generation quality of continuous diffusion models for text.
  • The adaptive noise schedule effectively balances denoising difficulty across time steps and token positions, leading to more robust text generation and revealing that tokens at earlier positions are recovered faster.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.