Reproduce “Efficient Low-rank Multimodal Fusion with Modality-Specific Factors” on OpenResearch

About this paper

Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency

Low-rank Multimodal Fusion (LMF) is introduced by researchers at Carnegie Mellon University to efficiently model interactions across multiple data modalities, overcoming the exponential complexity of prior tensor-based methods. This approach significantly reduces model parameters and computation time while achieving superior or competitive performance on sentiment analysis, emotion recognition, and speaker trait recognition tasks.

Key insights

• Complex inter-modal interactions can be effectively captured through low-rank approximations of the weight tensor, achieving efficiency without compromising expressive power.
• A relatively low rank (small 'r' value) is often sufficient to capture crucial multimodal dependencies, indicating that most inter-modal interactions are parsimonious.
• The low-rank decomposition acts as a beneficial inductive bias, reducing overfitting and leading to more robust models compared to full-rank tensor approaches.

Summary by alphaXiv.