Reproduce “Factorized Contrastive Learning: Going Beyond Multi-view Redundancy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Paul Pu Liang, Zihao Deng, Martin Ma, James Zou, Louis-Philippe Morency, Ruslan Salakhutdinov
Researchers from Carnegie Mellon, University of Pennsylvania, and Stanford developed FACTORCL, a self-supervised multimodal learning method that explicitly disentangles and leverages both shared and modality-unique task-relevant information. This approach achieved state-of-the-art performance across six real-world multimodal benchmarks and demonstrated an average improvement of 13.6% when incorporating mechanisms to remove task-irrelevant information.
Key insights
- Explicitly factorizing representations into shared and modality-unique components is essential for effectively leveraging all task-relevant information in complex multimodal scenarios.
- Minimizing mutual information upper bounds is crucial for removing task-irrelevant information, ensuring the learned representations are not only sufficient but also necessary.
- Carefully constructed multimodal augmentations can serve as a powerful proxy for task relevance in self-supervised settings, guiding models to learn both shared and unique task-specific features.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.