Reproduce “Towards a Multidimensional Evaluation Framework for Empathetic Conversational Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aravind Sesagiri Raamkumar, Siyuan Brandon Loh
A multidimensional framework is proposed for evaluating empathetic conversational systems beyond BLEU, perplexity, and lexical diversity. Tests on EmpatheticDialogues show that general-purpose and fine-tuned LLMs display stronger measured empathy behaviors than dedicated dialogue models, while conventional reference-based metrics often rank those systems lower.
Key insights
- Empathy is better represented as a profile of observable capabilities than as one undifferentiated score: a response may acknowledge emotion without interpreting the problem or encouraging further disclosure.
- The rankings from empathy-focused assessments differ sharply from rankings based on BLEU, perplexity, and Distinct-1, showing that linguistic similarity and diversity are insufficient proxies for empathetic quality.
- Fine-tuning Vicuna on empathetic conversations improves its measured behavior across categories; mirroring, for example, rises from 78.52% to 94.08% under the LLM-based evaluator.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.