Reproduce “Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models” on OpenResearch

About this paper

Mingrui Wu, Jiayi Ji, Oucheng Huang, Jiale Li, Yuhang Wu, Xiaoshuai Sun, Rongrong Ji

This research from Xiamen University developed R-Bench, a new benchmark to evaluate relationship hallucinations in Large Vision-Language Models, revealing that these models struggle more with relationship hallucinations than object hallucinations and perform poorly on fine-grained relational tasks. The study attributes these issues to data distribution biases and models' over-reliance on common sense over visual input.

Key insights

• Relationship hallucinations are more prevalent and severe in current LVLMs compared to object hallucinations, indicating a fundamental limitation in complex scene understanding.
• LVLMs demonstrate poor fine-grained local comprehension, struggling significantly with accurately identifying relationships between specific, localized objects.
• The long-tail distribution of relationship words in training data and specific relationship co-occurrence patterns (relationship-relationship, subject-relationship, relationship-object) contribute to these hallucinations.
• LVLMs tend to prioritize common-sense knowledge over visual evidence and struggle with sophisticated spatial and contextual reasoning, leading to counterfactual and illusion hallucinations.

Summary by alphaXiv.