Reproduce “Multimodal Conversational AI: A Survey of Datasets and Approaches” on OpenResearch

About this paper

Anirudh Sundar, Larry Heck

Sundar and Heck from Georgia Institute of Technology present a comprehensive survey defining multimodal conversational AI, formally formulating its objective, and taxonomizing its core challenges. The work evaluates existing datasets and state-of-the-art approaches, identifying limitations in current systems' ability to handle complex human-like dialogue, and emphasizes multimodal co-learning as a key future direction.

Key insights

• Achieving human-level conversational intelligence necessitates moving beyond unimodal systems to integrate a rich tapestry of senses and expressions.
• Existing multimodal datasets and approaches often oversimplify conversational dynamics, falling short in capturing implicit meanings, diverse speech acts, and long-term dialogue dependencies.
• A structured taxonomy focusing on representation, fusion, translation, alignment, and co-learning provides a critical framework for advancing the field.
• Multimodal co-learning presents a particularly promising direction for future research, leveraging resource-rich modalities to improve understanding in resource-poor ones.

Summary by alphaXiv.