Reproduce “Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning” on OpenResearch

About this paper

Duy A. Nguyen, Abhi Kamboj, Minh N. Do

Robult is a framework designed for robust multimodal learning that addresses both missing data modalities and limited labeled data by employing an information-theoretic approach based on Partial Information Decomposition. It utilizes novel soft Positive-Unlabeled (PU) contrastive and latent reconstruction losses to achieve superior performance across various sentiment analysis and classification benchmarks.

Key insights

• Explicitly leveraging Partial Information Decomposition (PID) to optimize for both redundant and unique information across modalities enhances robustness against missing data.
• Aligning unimodal latent representations with a fused, complete representation through a soft PU contrastive loss enables effective knowledge transfer and robust inference when modalities are absent.
• Preserving modality-specific unique information via a latent reconstruction loss is crucial for maintaining the distinct contributions of each modality, particularly in semi-supervised and single-modality inference settings.

Summary by alphaXiv.