Reproduce “MMTM: Multimodal Transfer Module for CNN Fusion” on OpenResearch

About this paper

Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L. Iuzzolino, Kazuhito Koishida

Microsoft-led research introduces the Multimodal Transfer Module (MMTM), a plug-and-play neural network component designed to effectively fuse information from different sensory modalities within CNNs. MMTM consistently improved performance in dynamic hand gesture recognition (e.g., 93.51% on EgoGesture), audio-visual speech enhancement (PESQ of 2.73), and human action recognition (91.99% on NTU-RGBD), often surpassing late fusion and prior intermediate fusion methods.

Key insights

• The global receptive field information derived from the squeeze operation is crucial for effective multimodal fusion, even when feature map spatial dimensions are initially aligned.
• Channel-wise gating mechanisms are essential for adaptively emphasizing relevant features and suppressing less important ones across modalities, outperforming simple summation-based fusion.
• Significant performance improvements stem from cross-modal interaction and intermediate "slow fusion," rather than solely from modality-specific feature recalibration.

Summary by alphaXiv.