Reproduce “Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gating” on OpenResearch

About this paper

Zhe Cheng, Wenyu Chen, Fode Zhang, Dehuan Shen

Causal Route Gating (CRG) is a training-free method designed to mitigate hallucinations in Large Vision-Language Models (LVLMs) by selectively gating internal attention head pathways based on their causal contribution to a decision. The approach reduced hallucination errors by up to 7.0% F1 score on POPE benchmarks and enhanced visual grounding across several evaluation metrics.

Key insights

• Hallucinations in LVLMs often stem from 'route competition' within attention heads, where textual priors causally dominate the final decision even when visual evidence suggests otherwise.
• Decision-aligned causal effects (CRE) are more effective for diagnosing and intervening on hallucination-driving pathways than attention-based proxies (like VAR), which only measure attention mass and not the actual impact on the output.
• Fine-grained, head-internal intervention, specifically by identifying and gating problematic textual pathways that conflict with visual cues, enables targeted hallucination mitigation without indiscriminately suppressing beneficial information.

Summary by alphaXiv.