Reproduce “HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuiyuan Wang, Zhixian Zhao, Hongfei Xue, Chengyou Wang, Shuai Wang, Hui Bu, Xin Xu, Lei Xie
HumDial-EIBench introduces a benchmark for evaluating emotional intelligence in Audio Language Models (ALMs) using human-recorded multi-turn dialogues. It reformulates subjective tasks into objective multiple-choice questions and incorporates a specific task for detecting acoustic-semantic conflicts, revealing a prevalent "text-dominance bias" where models struggle to integrate conflicting multimodal signals.
Key insights
- Current ALMs exhibit a widespread "text-dominance bias," over-relying on text semantics and struggling to integrate conflicting acoustic emotional cues, behaving more like "transcribe-then-understand" systems.
- A "structural decoupling" exists between textual and acoustic empathy in ALMs, where models can generate expressively voiced responses without necessarily possessing deep semantic empathetic understanding.
- LLM-as-a-judge evaluations for complex attributes like empathy show significant scoring variance, highlighting their instability for objective assessment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.