Reproduce “Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Models” on OpenResearch

About this paper

Vy Nguyen, Ziqi Xu, Jeffrey Chan, Estrid He, Feng Xia, Xiuzhen Zhang

RMIT University researchers developed Aspect-Based Causal Abstention (ABCA), a framework allowing Large Language Models to proactively assess their internal knowledge before generating an answer, thereby reducing factual errors. This method achieved state-of-the-art performance, including a 0.964 accuracy on unanswerable questions on TruthfulQA with GPT-4.1, by providing principled reasons for abstention.

Key insights

• Proactively analyzing the internal diversity of LLM parametric knowledge through causally-conditioned 'aspects' enables pre-generation abstention and reduces hallucinations.
• Modeling LLM reasoning as a heterogeneous Structural Causal Model (SCM) and using AIPW for effect estimation allows for a principled understanding of internal knowledge reliability.
• A dual-agent system effectively discovers and validates causally relevant aspects, while Centroid Angular Deviation analysis provides a robust mechanism for deciding when to abstain based on conflicting or insufficient internal knowledge.

Summary by alphaXiv.