Reproduce “MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daiqing Wu, Dongbao Yang, Jiashu Yao, Hongrui Zhang, Can Ma, Yu Zhou, Sicheng Zhao
Researchers from Tsinghua University and Nankai University introduced the Emotion Statement Judgement (ESJ) task, a scalable INSETS annotation pipeline, and the MVEI benchmark to assess visual emotional intelligence in Multimodal Large Language Models (MLLMs). Their optimized MLLM, EmObserver, achieved 86.23% accuracy on MVEI and generalized robustly to other benchmarks, demonstrating enhanced affective reasoning capabilities.
Key insights
- Visual emotional intelligence is multifaceted, encompassing sentiment polarity, emotion interpretation, scene context, and perception subjectivity, with existing MLLMs showing uneven capabilities across these dimensions.
- Dedicated multi-stage training tailored to the ESJ task significantly enhances an MLLM's affective reasoning, leading to improved accuracy and generalization beyond direct supervision.
- The task design (ESJ), which allows for natural language expression while constraining outputs, effectively fosters transferable affective learning in MLLMs by addressing the limitations of narrow and fixed-category AICA paradigms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.