Reproduce “CANDI: Hybrid Discrete-Continuous Diffusion Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Patrick Pynadath, Jiaxin Shi, Ruqi Zhang
CANDI keeps token positions clean while adding Gaussian noise to others, aligning two corruption processes so discrete conditioning and continuous updates coexist. At vocabulary size 50,000, 50% identity corruption yields 0.0029 competing coordinates above the correct one. On OpenWebText, CANDI’s entropy–perplexity frontier beats MDLM’s through 32 NFE, matches it at 64, and beats DUO’s below roughly 40 generative perplexity.
Key insights
- Continuous diffusion's struggle with discrete data, especially large vocabularies, is due to "temporal dissonance," a mismatch in how noise affects discrete token identity and continuous signal rank.
- At noise levels suitable for learning discrete dependencies, continuous signal degradation is trivial; conversely, when continuous signals are sufficiently degraded, discrete token identifiability is lost.
- Decoupling and independently controlling these corruption mechanisms allows CANDI to simultaneously learn discrete conditional structures and continuous gradients, leveraging the strengths of both paradigms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.