Reproduce “On the Role of Discreteness in Diffusion LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqi Jin, Bin Wang, Xiang Lin, Lidong Bing, Aixin Sun
Researchers from MiroMind AI and Nanyang Technological University introduce a conceptual framework for Diffusion Language Models (DLMs) to analyze challenges in applying continuous diffusion processes to discrete language. The work identifies that uniform noise corruption leads to uneven information loss, termed "frequency collapse," and that token-wise marginal training often results in a "marginal trap" where local predictions lack global coherence.
Key insights
- No single current DLM approach fully satisfies all five essential properties (D1-D3, L1-L2) simultaneously, revealing inherent trade-offs in their design.
- Uniform noise schedules, such as masking, lead to uneven information loss in text, where specific, high-impact tokens lose recoverable information much faster, resulting in "frequency collapse" in less constrained positions.
- Token-wise marginal training objectives in DLMs fail to explicitly model joint linguistic constraints, creating a "marginal trap" where locally plausible but globally inconsistent sequences can be generated during parallel decoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.