Reproduce “Generative Frontiers: Why Evaluation Matters for Diffusion Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Patrick Pynadath, Jiaxin Shi, Ruqi Zhang
The authors compare diffusion language models using generative perplexity–entropy frontiers traced across sampling temperatures, rather than single settings. KL divergence relates cross-entropy to sequence entropy, approximated here by unigram entropy. On OpenWebText, Duo leads at matched natural-language entropy, while CANDI leads at matched likelihood. These comparisons depend on unigram entropy ranking sequence entropies.
Key insights
- Direct comparisons of ELBOs between dLLMs with different latent variable structures are unreliable due to the intractable KL divergence term.
- Generative perplexity is highly sensitive to minor unigram entropy changes because they are inherently linked through the components of KL divergence.
- Single-point evaluations of generative quality are ambiguous, as dLLMs define a family of distributions, and one model might appear superior on one metric while being inferior on another, depending on inference settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.