Reproduce “RE-LLM: Refining Empathetic Speech-LLM Responses by Integrating Emotion Nuance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jing-Han Chen, Bo-Hao Su, Ya-Tse Wu, Chi-Chun Lee
The paper presents RE-LLM, which combines speech features with a frozen wav2vec 2.0 emotion encoder and auxiliary emotion objectives to generate empathetic responses and follow-up questions. On single-turn ESD examples, it scores 1.847 for Emotional Reaction and 1.206 for Exploration, versus 1.730 and 1.098 for fine-tuned BLSP-Emo. IEMOCAP Exploration is not significantly higher, and human empathy was not tested.
Key insights
- Systematically incorporating rich emotion embeddings from speech, combined with multi-perspective auxiliary learning, enhances an LLM's capacity for both emotional reflection and exploration.
- Emotional exploration, characterized by generating probing questions, is a crucial component of empathy that can be significantly improved by leveraging nuanced speech emotion information.
- The use of a dedicated emotion encoder along with multi-task learning (categorical and dimensional emotion prediction) helps the LLM better perceive and utilize subtle emotional cues from speech.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.