Reproduce “DialFact: A Benchmark for Fact-Checking in Dialogue” on OpenResearch

About this paper

Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong

Researchers from Carnegie Mellon University and Salesforce AI Research introduce DialFact, the first comprehensive benchmark dataset for fact-checking in dialogue. The dataset defines and evaluates three sub-tasks: verifiable claim detection, evidence retrieval, and claim verification, demonstrating that existing fact-checking models perform poorly and that a weakly-supervised synthetic data approach consistently improves performance across these tasks.

Key insights

• Fact-checking in dialogue requires specific models that can handle colloquialisms, coreferences, and the distinction between verifiable facts and opinions, as traditional models perform poorly.
• Effective evidence retrieval in dialogue contexts relies heavily on entity linking (e.g., via WikiAPI) and the incorporation of dialogue history, outperforming dense passage retrieval (DPR) methods.
• Weakly-supervised synthetic data, generated through techniques like negation and substitution, can substantially improve the performance of models for fact-checking in dialogue, demonstrating a data-efficient training strategy.

Summary by alphaXiv.