Reproduce “Rényi Divergence and Kullback-Leibler Divergence” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tim van Erven, Peter Harremoës
This work provides a comprehensive and rigorously detailed reference for R´enyi divergence, consolidating its mathematical properties and extending its definition and fundamental inequalities to continuous probability spaces. It precisely characterizes the extended orders (0, 1, and ), generalizes the Pythagorean inequality, and clarifies the measure's limitations with specific counterexamples.
Key insights
- R´enyi divergence serves as a parametrized family encompassing KL divergence, with extended orders (0, 1, ) having precise closed-form expressions that clarify their interpretations (e.g., absolute continuity, worst-case regret).
- The Pythagorean inequality can be generalized to any order (0, ), providing new geometric insights into information projection beyond KL divergence.
- Despite its broad utility, R´enyi divergence has specific limitations, such as lack of convexity in the first argument for > 1 or metric properties for =1/2, which must be carefully considered during application.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.