Reproduce “When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinjin Cao, Zhiyang Chen, Zijun Wang, Liyuan Ma, Weijian Luo, Guojun Qi
Researchers at Westlake University developed Cross-Modal Guidance (CMG), a training-free decoding method that addresses language bias-induced hallucinations in Vision-Language Models by strategically manipulating internal attention weights. This approach achieved an 85.4% overall accuracy on the POPE benchmark using LLaVA-v1.5-7B, outperforming existing contrastive decoding methods, and demonstrated improved perception scores on the MME benchmark across various model scales.
Key insights
- Language bias, characterized by diminishing attention to image tokens in deeper layers and longer sequences, is a root cause of hallucinations in VLMs.
- Manipulating internal attention weights can directly counteract language bias, making VLMs prioritize visual context during text generation.
- A carefully calibrated 'amateur model' that is sufficiently weakened, rather than completely dysfunctional, is essential for effective contrastive decoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.