Reproduce “LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jennifer A Bishop, Qianqian Xie, Sophia Ananiadou
The National Centre for Text Mining introduced LongDocFACTScore, a framework and dataset for evaluating the factual consistency of abstractive summaries for long documents. It achieves higher correlations with human judgments than existing methods by efficiently assessing summary sentences against relevant snippets from the entire source document, addressing the limitations of prior truncation-based approaches.
Key insights
- Applying existing automatic factuality metrics directly to long documents with truncation results in poor correlation with human judgments, demonstrating their inadequacy.
- An intelligent, snippet-based sampling approach allows for accurate, fine-grained factual consistency assessment across the entire source document without token limit constraints.
- The LongDocFACTScore framework achieves superior performance on long documents while maintaining competitive computational efficiency compared to existing methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.