Reproduce “Treble Counterfactual VLMs: A Causal Approach to Hallucination” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao
This work introduces a causal framework to analyze and mitigate hallucination in Vision-Language Models (VLMs) by identifying and counteracting "spurious modality shortcuts." The approach quantifies unintended direct influences of individual modalities using Natural Direct Effect (NDE) estimation and applies a test-time intervention to adjust VLM representations, leading to significant reductions in hallucination across benchmarks and VLM architectures.
Key insights
- VLM hallucination arises from unintended direct influences or "spurious modality shortcuts" from either the vision or text modality, bypassing proper multi-modal fusion.
- Natural Direct Effect (NDE) estimation effectively quantifies these direct, ungrounded influences within the VLM's intermediate representations.
- A test-time intervention, by adjusting these representations based on estimated NDEs, efficiently mitigates hallucination without requiring model re-training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.