Reproduce “Information-Theoretic Discrete Diffusion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Moongyu Jeon, Sangwoo Shin, Dongjae Jeon, Albert No
The authors derive exact information-theoretic identities showing that optimal masked-token reconstruction loss tracks information lost along a corruption path and integrates to sequence negative log-likelihood under full-noise conditions. They propose a time-free estimator that samples visible-token subsets; on HellaSwag, its variance was 11.57 versus 70.97 for time-integral estimates at 128 samples, demonstrating lower Monte Carlo variance in tested settings.
Key insights
- Denoising Score Entropy (DSE) and Denoising Cross-Entropy (DCE) losses, commonly used in discrete diffusion, function as exact estimators for the negative log-likelihood rather than merely variational upper bounds, provided the score network is optimal.
- A rigorous information-theoretic framework can be extended to discrete diffusion models, providing discrete analogs (I-MDSE, I-MDCE) to the I-MMSE identity foundational for continuous diffusion.
- Likelihood and likelihood ratio estimations in discrete diffusion can be significantly improved in efficiency and stability through time-free and coupled Monte Carlo estimators, offering reduced variance and computational advantages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.