Reproduce “Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiyue Zhang, David Wan, Mohit Bansal
An investigation into extractive summarization reveals that approximately 30% of system-generated summaries contain broad unfaithfulness problems, challenging the assumption of their inherent fidelity. Researchers from UNC Chapel Hill developed a new error typology and an automatic metric, EXTEVAL, which effectively identifies these issues, outperforming existing evaluation tools.
Key insights
- Extractive summarization is not inherently faithful; a substantial portion (30.3%) of system-generated extractive summaries exhibit at least one type of unfaithfulness.
- The placement of extracted sentences significantly impacts faithfulness, with systems extracting sentences from the middle of documents often producing more coreference and discourse linking issues.
- Most existing automatic faithfulness metrics demonstrate poor correlation with human judgments of broad unfaithfulness in extractive summaries, indicating their inadequacy for this task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.