Reproduce “CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tao Feng, Lizhen Qu, Xiaoxi Kang, Gholamreza Haffari

CausalScore is an automatic, reference-free metric that assesses the relevance of responses in open-domain dialogue systems by quantifying the causal strength between dialogue history and the response. It achieves significantly higher correlation with human judgments for relevance and specificity compared to state-of-the-art baselines, and introduces the CGDIALOG+ dataset with human-annotated causal relations.

Key insights

  • Highly relevant responses in a dialogue system exhibit strong, quantifiable causal relationships with specific utterances in the preceding conversation.
  • Conditional dependence testing is crucial for accurately measuring causal strength and discerning true relevance, as demonstrated by its significant impact in ablation studies.
  • Relying on high-quality human annotations for causal relations, as provided in the CGDIALOG+ dataset, is critical for training robust and effective causal inference models for dialogue evaluation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.