Reproduce “Evaluating Evidence Attribution in Generated Fact Checking Explanations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rui Xing, Timothy Baldwin, Jey Han Lau
Researchers at The University of Melbourne introduced a "Citation Masking and Recovery" protocol to evaluate evidence attribution in LLM-generated fact-checking explanations. The study found that GPT-4 produced the most accurate attributions and served as the most reliable automated assessor, achieving an F1 score of up to 0.74, while highlighting that human-curated evidence consistently improved attribution quality in explanations.
Key insights
- LLMs are more suitable for automating attribution assessment than NLI models due to their ability to leverage contextual information, with GPT-4 demonstrating the highest agreement with human judgments.
- The source of evidence significantly impacts attribution quality; explanations generated using human-curated evidence consistently achieve higher and more reliable attribution fidelity than those based on machine-selected evidence.
- Even the best-performing LLMs for explanation generation, like GPT-4, still have substantial room for improvement in consistently and fully attributing all statements in an explanation to their correct sources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.