Reproduce “Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning” on OpenResearch

About this paper

Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, Xun Yang

Researchers from the University of Science and Technology of China and Nanyang Technological University introduced CA-MER, a benchmark to evaluate multimodal large language models (MLLMs) under emotion conflict conditions. They also developed MoSEAR, a framework that mitigates observed audio modality bias by improving MLLM attention and learning, achieving up to a 9.4% F1 score improvement on emotion recognition (MER2023) and reducing the performance gap in conflict scenarios from 12% to 6% on the new CA-MER benchmark.

Key insights

• Existing emotion MLLMs exhibit a systematic audio modality bias, performing significantly worse (e.g., 12% lower accuracy for Emotion-LLaMA) on video-aligned emotion conflicts compared to audio-aligned ones on the CA-MER benchmark.
• The extreme disparity in token counts between video and audio inputs is a key factor contributing to the observed audio over-reliance in current MLLMs.
• A carefully designed, selective attention reallocation mechanism can effectively mitigate modality bias in frozen MLLM backbones during inference, avoiding the performance trade-offs observed in prior attention-shifting methods.

Summary by alphaXiv.