Reproduce “HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses through Reasoning MLLMs” on OpenResearch

About this paper

Zheng Qin, Ruobing Zheng, Yabing Wang, Tianqi Li, Yi Yuan, Jingdong Chen, Le Wang

This work introduces HumanSense, a comprehensive benchmark for evaluating multimodal large language models (MLLMs) on human-centered perception, implicit contextual understanding, and empathetic, context-aware response generation. The study reveals that current MLLMs significantly lag human performance in complex interactive scenarios and demonstrates that both reinforcement learning and prompt engineering can enhance their reasoning capabilities.

Key insights

• Multimodal input, particularly raw audio integrated with video, is crucial for MLLMs to achieve higher-level human understanding and generate more appropriate, context-aware responses.
• The primary bottleneck for state-of-the-art MLLMs in human-centered tasks is a deficiency in high-level, long-context reasoning rather than solely low-level perception.
• Both reinforcement learning with modality-progressive training and targeted prompt engineering can effectively enhance MLLMs' reasoning abilities, substantially improving their performance on complex human-centered interaction tasks.

Summary by alphaXiv.