Reproduce “TRUE: Re-evaluating Factual Consistency Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor and 2 more
Researchers from Google Research, Tel Aviv University, and Meta AI systematically re-evaluated factual consistency metrics, standardizing 11 diverse datasets and proposing an example-level meta-evaluation protocol using ROC AUC. The study found that large-scale Natural Language Inference (NLI) and Question Generation and Answering (QG-QA) methods achieved superior performance, especially when combined, while also highlighting challenges with long inputs and subtle inconsistencies.
Key insights
- Large-scale NLI and QG-QA methods demonstrate superior performance for factual consistency evaluation, with an ensemble of these approaches yielding the strongest results.
- The example-level ROC AUC meta-evaluation provides a more actionable and interpretable measure of a metric's quality compared to traditional system-level correlations.
- All evaluated metrics show a consistent degradation in performance when processing grounding texts longer than 200 tokens, indicating a significant challenge in handling long contexts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.