Reproduce “Emotional Cues Extraction and Fusion for Multi-modal Emotion Prediction and Recognition in Conversation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoxiang Shi, Ziqi Liang, Jun Yu
Researchers at the University of Science and Technology of China introduced a multi-modal framework for conversational emotion prediction and recognition, integrating word-level emotional cues and modality-specific audio enhancements with a refined two-step fusion strategy. This framework achieved superior performance in multi-modal emotion prediction, boosting UAR by 1.74% on IEMOCAP, and delivered competitive results for emotion recognition.
Key insights
- Explicitly modeling word-level emotional relationships, rather than solely relying on utterance-level context, significantly improves the perception of emotional cues within text.
- Designing modality-specific enhancements, such as a dedicated prosody enhancement for audio, effectively extracts and amplifies unique emotional information from each modality.
- A two-step multi-modal fusion strategy that incorporates spectral domain features from mel-spectrograms and leverages cross-modal attention through bridge vectors enables more comprehensive and complementary information capture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.