Reproduce “Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingfei Han, Haihong Hao, Jinxing Zhou, Zhihui Li, Yuhui Zheng, Xueqing Deng, Linjie Yang, Xiaojun Chang
A method for reducing hallucinations in Vision-Language Models leverages internal self-consistency, using simple binary questions to identify and correct inconsistencies in detailed generative responses. This approach decreases the hallucination rate on the AMBER benchmark from 36.4% to 10.3% for LLaVA-v1.5-7B, improving factual grounding without external supervision or human annotations.
Key insights
- VLMs tend to provide more factually accurate answers to simple binary questions compared to complex, open-ended generative tasks.
- This "behavioral consistency gap" between simple queries and elaborate generations serves as a reliable, internal self-supervisory signal for detecting hallucinations.
- Negative binary responses from the VLM itself are highly precise indicators of hallucinations (e.g., 94% for object-existence claims).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.