Reproduce “OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue” on OpenResearch

About this paper

Xuelong Geng, Qijie Shao, Hongfei Xue, Shuiyuan Wang, Hanke Xie, Zhao Guo, Yi Zhao, Guojian Li and 15 more

The OSUM-EChat system developed by the Audio, Speech and Language Processing Group at Northwestern Polytechnical University enhances end-to-end empathetic spoken chatbots by integrating an understanding-driven training strategy and a linguistic-paralinguistic dual think mechanism. It achieved a GPT-4 score of 72.0 on a new EChat-eval benchmark for multi-label empathy, demonstrating improved empathetic responsiveness and efficient speech understanding without relying on massive, proprietary datasets.

Key insights

• Effective empathetic spoken dialogue systems can be developed in resource-limited environments by leveraging knowledge transfer from pre-trained speech understanding models.
• Explicit, understanding-driven reasoning about both linguistic and paralinguistic cues through a dual thinking mechanism significantly enhances a model's ability to generate contextually coherent and empathetic responses.
• An open-source ecosystem, including models, codebases, and specialized datasets/benchmarks, is crucial for fostering broader academic advancement and community-driven research in empathetic AI.

Summary by alphaXiv.