Reproduce “HalluGraph: Auditable Hallucination Detection for Legal RAG Systems via Knowledge Graph Alignment” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Valentin Noël, Elimane Yassine Seidou, Charly Ken Capo-Chichi, Ghanem Amari
Devoteam researchers developed HalluGraph, an auditable framework for detecting hallucinations in legal Retrieval-Augmented Generation (RAG) systems by aligning knowledge graphs extracted from AI-generated responses with source documents. The method achieved an AUC of 0.94 on Legal Contract QA and 0.84 on Legal Case QA, significantly outperforming semantic similarity baselines.
Key insights
- Structural and relational fidelity, rather than just semantic similarity, is essential for robust hallucination detection in entity-rich, high-stakes domains like legal text.
- The combination of entity grounding and relation preservation through the Composite Fidelity Index (CFI) provides a superior and more precise measure of factual accuracy.
- HalluGraph performs optimally in environments with longer, entity-dense documents, such as legal contracts and case opinions, where meaningful knowledge graphs can be constructed.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.