Reproduce “Concepts' Information Bottleneck Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Karim Galliamov, Syed M Ahsan Kazmi, Adil Khan, Adín Ramírez Rivera
Researchers from the University of Amsterdam, University of West of England, University of Hull, and University of Oslo developed Concepts’ Information Bottleneck Models (CIBMs), integrating the Information Bottleneck principle into Concept Bottleneck Models (CBMs) to mitigate concept leakage and enhance predictive accuracy. The proposed CIBMs consistently achieved higher class prediction accuracy, significantly reduced concept leakage, and improved the reliability of test-time interventions across various CBM architectures and datasets.
Key insights
- Directly penalizing I(X; C) at the concept layer effectively curbs spurious information encoding in concepts, leading to significant reductions in concept leakage without complex architectural alterations.
- A theoretically-grounded, information-theoretic regularization approach can simultaneously improve predictive performance, explanation faithfulness, and the reliability of test-time interventions in CBMs.
- The PAC-Bayes theoretical analysis indicates that CIBMs can achieve a tighter generalization bound and lower true risk, suggesting a more robust learning mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.