Reproduce “Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learning” on OpenResearch

About this paper

Kaiqi Chen, Yong Lee, Harold Soh

Researchers at NUS developed Multi-Modal Mutual Information (MuMMI) training, a self-supervised approach that enables deep reinforcement learning agents to learn robust world models from multiple, potentially unreliable, sensory inputs. This method consistently outperforms existing techniques in multi-modal environments and demonstrates strong resilience to missing data, achieving higher average scores and faster learning.

Key insights

• Maximizing mutual information between individual modalities and a shared latent space is more effective for learning robust multi-modal representations than standard reconstruction-based objectives.
• Explicitly encouraging consistency between *individual modality experts* and the fused latent code significantly improves the robustness of multi-modal fusion, outperforming methods that only ensure consistency for the aggregated output.
• A Product-of-Experts (PoE) approach, combined with MI maximization, offers a principled and robust mechanism for integrating diverse sensory inputs and handling data scarcity in multi-modal DRL.

Summary by alphaXiv.