Reproduce “A Generalized Information Bottleneck Method: A Decision-Theoretic Perspective” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Akira Kamatsuka, Takahiro Yoshida

Researchers from Shonan Institute of Technology and Nihon University developed a generalized Information Bottleneck (IB) method that provides a decision-theoretic interpretation for the utility of compressed data, mapping specific H-mutual information measures to the Expected Value of Sample Information (EVSI). Their work includes an alternating optimization algorithm that illustrates how distinct utility functions, such as Shannon mutual information or EVSI for squared-error loss, lead to different optimal data representations for tailored compression.

Key insights

  • A specific class of H-mutual information (satisfying CV and AVG conditions) provides a direct, decision-theoretic interpretation of utility in the IB framework by equating it with the Expected Value of Sample Information (EVSI).
  • The choice of utility function, anchored in a specific decision-theoretic context (e.g., squared-error loss), directly influences the optimal compression strategy and the resulting compressed representation p_T|X.
  • The proposed generalized framework is consistent with the original IB method, as its algorithm's update rules simplify to those of classical IB when Shannon mutual information is used as the utility measure.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.