Reproduce “Dependency, Compression, and Synergy: A Unified Information-Theoretic View of Multimodal Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liangjian Wen, Linjie Li, Jiang Duan, Yong Dai, Jianzhuang Liu, Zhao Kang
A survey of 170 studies frames multimodal learning through three increasingly detailed information-theoretic questions: how modalities depend on one another, which information should be retained under compression, and whether useful information is redundant, unique, or synergistic. Its Generalized Multimodal Information Lagrangian (GMIL) places mutual-information alignment, information-bottleneck fusion, and PID-guided methods in one framework, while highlighting unresolved challenges in estimation, scaling, and foundation-model pretraining.
Key insights
- Maximizing mutual information can align modalities effectively, but by itself does not ensure minimal representations, suppress nuisance information, or preserve task-relevant information unique to one modality.
- Information Bottleneck objectives are associated with better robustness to missing modalities, noise, distribution shift, and communication constraints, while PID reveals why shared-information objectives can fail on synergy-dependent tasks such as XOR-style prediction.
- Compression cannot increase total encoded PID information under the stated encoder assumptions, but it does not specify whether redundancy, uniqueness, or synergy will be discarded; that behavior depends on the estimator and objective.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.