Reproduce “Meta-DiffuB: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yun-Yen Chuang, Hung-Min Hsu, Kevin Lin, Chen-Sheng Gu, Ling Zhen Li, Ray-I Chang, Hung-yi Lee
Meta-DiffuB introduces a contextualized noise-scheduling strategy for sequence-to-sequence text diffusion models, leveraging a meta-exploration framework to learn adaptive noise levels. This approach achieved state-of-the-art performance across various text generation benchmarks and enhanced text diversity, demonstrating its 'plug-and-play' utility.
Key insights
- Applying learnable, contextualized noise scheduling based on input sentence characteristics can significantly improve text diffusion model performance and adaptability.
- A meta-exploration framework enables a scheduler to learn optimal, dynamic noise policies by rewarding the performance improvements it brings to the main generative model, even for non-differentiable noise scheduling processes.
- The learned noise strategy strategically applies less noise to semantically 'harder' sentences and more noise to 'easier' sentences, enhancing both generation quality and diversity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.