Reproduce “Fast and Accurate Factual Inconsistency Detection Over Long Documents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Barrett Martin Lattimer, Patrick Chen, Xinyuan Zhang, Yi Yang
ASAPP and Abstractive Health developed SCALE, an NLI-based model for fast, accurate, and explainable factual inconsistency detection in LLM-generated content, particularly over long source documents. This work also introduces ScreenEval, a new dataset for evaluating inconsistency in long-form dialogues.
Key insights
- Chunking long source documents into larger, contextual segments for NLI evaluation significantly improves accuracy and calibration of factual inconsistency detection compared to sentence-level comparisons.
- Providing sufficient context to the NLI model through effective chunking leads to better-calibrated inconsistency scores, which are more reliable indicators of factual correctness.
- An efficient hierarchical search strategy can pinpoint supporting or contradicting evidence in long documents with substantial speed improvements, making explainability practical for real-time applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.