Reproduce “Self-supervised Learning from a Multi-view Perspective” on OpenResearch

About this paper

Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, Louis-Philippe Morency

Carnegie Mellon University researchers established an information-theoretic framework for self-supervised learning (SSL) from a multi-view perspective, explaining how SSL objectives extract task-relevant information while also introducing an inverse predictive learning objective to discard task-irrelevant information. The proposed objective empirically improved performance in both visual and cross-modal tasks, including a 47% accuracy on a professional Go move prediction task and consistently defeating 10 kyu amateur players on an online server.

Key insights

• Self-supervised learning can effectively extract task-relevant information from inputs, with the amount of potential loss bounded by the redundancy (ε_info) between the input and its self-supervised signal.
• Actively discarding task-irrelevant information is crucial for robust representations, but there exists an irreducible 'compression gap' (I(X; S|T)) of shared information not relevant to the downstream task.
• Contrastive and forward predictive learning primarily focus on extracting task-relevant information, while inverse predictive learning specifically addresses discarding task-irrelevant information.

Summary by alphaXiv.