Reproduce “CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models” on OpenResearch

About this paper

Neeraj Anand, Samyak Jha, Udbhav Bamba, Rahul Rahaman

CRoPS is a training-free framework that effectively mitigates hallucinations in Large Vision-Language Models by dynamically integrating a novel text-deficit hallucinated model with a visual-deficit model. The approach consistently reduces hallucination rates by approximately 15-25% on benchmarks like CHAIR for LLaVA-1.5 while maintaining fluency and improving general multimodal reasoning with minimal inference overhead.

Key insights

• Large Vision-Language Models' reliance on visual input diminishes significantly during later stages of text generation, making purely visual-token-perturbing methods less effective for hallucination mitigation over time.
• Hallucinations in LVLMs arise from multifaceted sources, requiring a comprehensive approach that moves beyond single-source mitigation strategies.
• A novel "text-deficit" hallucinated model, which removes visual tokens and crucial textual context, consistently produces a distinct contrastive signal crucial for effective and sustained hallucination suppression.

Summary by alphaXiv.