Reproduce “Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, Aimin Zhou
A detailed investigation shows that the CLIP visual encoder inherently contributes to object hallucinations in Large Vision-Language Models (LVLMs). A proposed fine-tuning method, leveraging a new benchmark and enhanced contrastive loss, improves CLIP's object hallucination detection from approximately 19% to 88% and reduces hallucinations in downstream LVLMs like LLaVA-1.5.
Key insights
- CLIP visual encoders inherently suffer from a significant object hallucination problem, with vanilla models correctly identifying the positive caption only ~19% of the time on the OHD-Caps benchmark.
- Targeted fine-tuning with counterfactual object-level negative samples and a specialized contrastive loss effectively mitigates object hallucinations in CLIP, demonstrating strong generalization to out-of-domain data.
- Improving the visual encoder directly translates to reduced object hallucinations in downstream LVLMs like LLaVA-1.5, without compromising their general multimodal capabilities on other tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.