Reproduce “DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Miriam Wanner, Benjamin Van Durme, Mark Dredze
Researchers at Johns Hopkins University developed DNDSCORE, a novel decontextualization-aware method for verifying the factuality of long-form text generated by Large Language Models, which systematically integrates claim decomposition and decontextualization. This approach enabled a more accurate assessment, leading to DNDSCORE achieving factuality scores up to 61.51% by explicitly using decontextualized context during verification.
Key insights
- Explicitly handling context through decontextualization and DNDSCORE significantly increases the proportion of verifiable claims in LLM-generated text, suggesting previous factuality scores might have been underestimated.
- DNDSCORE is robust to cases where decontextualization might introduce incorrect contextual information, still allowing the original atomic subclaim to be correctly verified.
- The joint DnD method efficiently integrates both processes, demonstrating that addressing their interplay from the outset can yield effective and flexible fact-checking inputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.