Reproduce “Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models” on OpenResearch

About this paper

Weihang Wang, Xinhao Li, Ziyue Wang, Yan Pang, Jielei Zhang, Peiyi Li, Qiang Zhang, Longwen Gao

A fine-grained, vision-centric benchmark, VHBench-10, was developed to diagnose distinct visual hallucination types in Large Vision-Language Models (LVLMs). VisionWeaver, a context-aware routing network, was then proposed to address these issues, achieving an 88.8% F1 score on POPE and a 54.3% accuracy on AutoHallusion, while enhancing general vision benchmark performance.

Key insights

• Different visual encoders exhibit distinct inductive biases and unique hallucination characteristics, confirming the hypothesis that their diverse training paradigms lead to specialized visual perception strengths and weaknesses.
• Fine-grained hallucination analysis is crucial for understanding specific visual perception deficiencies in LVLMs, moving beyond general error detection to targeted diagnosis.
• Dynamically routing and aggregating visual features from specialized experts based on global visual understanding is more effective than simple feature fusion or relying on a single encoder for hallucination mitigation.

Summary by alphaXiv.