Reproduce “AffectSeek: Agentic Affective Understanding in Long Videos under Vague User Queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhen Zhang, Yuhang Yang, Yunxiang Jiang, Yuhuan Lu, Haifeng Lu, Zheng Lian, Runhao Zeng, Xiping Hu
The research introduces AffectSeek, a multi-agent framework, and VQAU-Bench, a new benchmark, to address Vague-Query-driven video Affective Understanding (VQAU) in long, untrimmed videos. AffectSeek demonstrated superior performance in localizing affective moments, classifying emotions, and generating evidence-grounded rationales when guided by vague natural language queries, achieving an mIoU of 19.55 for localization and a Joint R@1 of 15.91 at tIoU at 0.3.
Key insights
- Agentic, multi-step reasoning is essential for interpreting vague affective queries, localizing sparse evidence in long videos, and generating reliable explanations, outperforming single-step inference models.
- A unified benchmark like VQAU-Bench is critical for evaluating models' ability to perform semantic-temporal-affective alignment, highlighting the limitations of models optimized for clip-level recognition.
- Integrating evidence-grounded rationales, derived from multimodal evidence and verified through a reflection mechanism, significantly enhances the interpretability and trustworthiness of affective understanding systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.