Reproduce “Information Estimation with Discrete Diffusion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alberto Foresti, Giulio Franzese, Pietro Michiardi

INFO-SEDD estimates KL divergence, mutual information, and entropy directly from high-dimensional discrete data using score models trained on continuous-time Markov-chain diffusion. It avoids continuous embeddings and performs well on synthetic benchmarks, while applications recover expected trends in summarization, DNA sequence analysis, and Ising-model entropy estimation.

Key insights

  • The conditional MI estimator can be more effective than the joint formulation when modeling the full sequence-label distribution is difficult; in DNA experiments it followed a classifier-based reference more closely.
  • MI estimates can reflect source-summary consistency, but they do not capture every human evaluation dimension and do not directly penalize hallucinations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.