Reproduce “VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhicheng Zhang, Weicheng Wang, Yongjie Zhu, Wenyu Qin, Pengfei Wan, Di Zhang, Jufeng Yang

Researchers from Nankai University and Kuaishou Technology developed VidEmo, an affective-tree reasoning framework, and Emo-CFG, a large-scale dataset, to enhance emotion understanding in Video Large Language Models. VidEmo integrates attribute perception, expression analysis, and high-level emotional reasoning, achieving state-of-the-art performance across 15 face perception tasks and outperforming leading commercial APIs.

Key insights

  • A structured, hierarchical 'affective-tree' reasoning approach, progressing from attributes to expressions to emotions, effectively enables models to achieve more cognitive and interpretable emotion understanding.
  • Reinforcement learning, guided by a 'Mixed Affective-Tree Reward' that measures tree edit distance against ground-truth rationales, is crucial for refining the generation of structurally explainable emotional captions.
  • High-quality, rationale-rich instructional data, verified through a 'committee voting' strategy by heterogeneous VideoLLMs, is essential for training models that can reason about and explain complex emotional states.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.