Reproduce “Structured IB: Improving Information Bottleneck with Structured Feature Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanzhe Yang, Youlong Wu, Dingzhu Wen, Yong Zhou, Yuanming Shi
The Structured Information Bottleneck (SIB) framework enhances Information Bottleneck Lagrangian methods by integrating multiple auxiliary encoders and a multi-stage training process. This approach consistently improves prediction accuracy and task-relevant information extraction, while often increasing parameter efficiency across datasets like MNIST and CIFAR-10.
Key insights
- Auxiliary features can theoretically augment the information content of the original feature regarding the target variable, provided certain conditions like feature independence are met (Theorem 1).
- A structured, multi-stage training regimen allows for the generation of complementary and non-redundant representations, enhancing the overall informativeness of the feature space.
- The combination of explicit feature independence enforcement and sequential training enables the framework to overcome limitations of single-feature IB methods and approximation errors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.