Reproduce “CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tao Feng, Lizhen Qu, Xiaoxi Kang, Gholamreza Haffari
CausalScore is an automatic, reference-free metric that assesses the relevance of responses in open-domain dialogue systems by quantifying the causal strength between dialogue history and the response. It achieves significantly higher correlation with human judgments for relevance and specificity compared to state-of-the-art baselines, and introduces the CGDIALOG+ dataset with human-annotated causal relations.
Key insights
- Highly relevant responses in a dialogue system exhibit strong, quantifiable causal relationships with specific utterances in the preceding conversation.
- Conditional dependence testing is crucial for accurately measuring causal strength and discerning true relevance, as demonstrated by its significant impact in ablation studies.
- Relying on high-quality human annotations for causal relations, as provided in the CGDIALOG+ dataset, is critical for training robust and effective causal inference models for dialogue evaluation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.