Reproduce “AEQ-Bench: Measuring Empathy of Omni-Modal Large Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuan Luo, Lewei Yao, Libo Zhao, Lanqing Hong, Kai Chen, Dehua Tao, Daxin Tan, Ruifeng Xu and 1 more
Researchers from The Hong Kong Polytechnic University, Huawei, and collaborators developed AEQ-Bench, a benchmark to measure empathy in Omni-Modal Large Models by evaluating both linguistic and paralinguistic cues in audio and text. The study found that while top-tier OLMs like GPT-4o and Qwen's Omni series achieve human-like empathetic responses and strong multimodal integration (e.g., GPT-4o with 0.91 normalized score for multimodal reliance), they still struggle with fine-grained paralinguistic expressiveness.
Key insights
- Multimodal integration is crucial for empathy, with audio-output OLMs generally outperforming text-only models in generating human-like empathetic responses.
- While top-tier OLMs can align with human judgments for coarse-grained empathy evaluation, they demonstrate unreliability in discerning fine-grained paralinguistic expressiveness.
- Textual captions of audio are not faithful substitutes for direct auditory perception in assessing paralinguistic empathy, often overlooking nuances and introducing biases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.