Reproduce “Evaluating the Factual Consistency of Abstractive Text Summarization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher
Salesforce Research developed a weakly-supervised, BERT-based method for evaluating the factual consistency of abstractive text summaries. This approach, named FactCC, outperformed models trained on general Natural Language Inference or fact-checking datasets and provided explainability by highlighting consistent and inconsistent text spans.
Key insights
- Factual consistency for abstractive summarization is a distinct task requiring specialized training, as general Natural Language Inference or fact-checking models perform poorly.
- Weakly-supervised data generation through rule-based transformations is effective and scalable for creating training data for complex NLP tasks like factual consistency checking.
- Providing model-generated explanations (highlights) improves human annotation efficiency and inter-annotator agreement for subjective tasks like consistency verification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.