Reproduce “Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuma Tsuta, Naoki Yoshinaga, Shoetsu Sato, Masashi Toyoda

Researchers at The University of Tokyo developed an automatic response evaluation method for open-domain dialogue systems that prioritizes the human interlocutor's perspective. Their method, trained via a Dialogue Continuity Prediction (DCP) task, achieved higher correlations with actual human engagement judgments for human-generated responses compared to evaluations based on third-party annotators.

Key insights

  • Evaluations from third-party annotators are poor proxies for actual human interlocutor judgments, highlighting a fundamental discrepancy in current evaluation practices.
  • Personalizing evaluation models to the specific human interlocutor (interlocutor awareness) is crucial for accurately predicting their subjective engagement with dialogue responses.
  • The Dialogue Continuity Prediction (DCP) task can serve as an effective, low-cost unsupervised method to train interlocutor-aware dialogue evaluators, especially for human-like responses.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.