Reproduce “Test-Time Hallucination Control in Large Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mehran Tamjidi, Hamidreza Dastmalchi, Ali Cheraghian, Mohammadreza Alimoradijazi, Aijun An, Hossein Rahmani
Test-Time Hallucination Mitigation (TTH) is a training-free decoding strategy that integrates visual grounding from a zero-shot Multi-Modal Classifier into Large Vision-Language Models to reduce object hallucinations. It improved factual accuracy, achieving, for instance, a reduction in the CHAIR_S score to 14.27 from 20.40 for LLaVA-1.5, while maintaining inference efficiency.
Key insights
- Object hallucinations often arise from low-confidence object predictions in LVLMs, where the model's logit distribution exhibits high entropy.
- Combining the generative fluency of LVLMs with the strong visual grounding capabilities of zero-shot MMCs like CLIP can effectively mitigate hallucinations without requiring model retraining.
- An adaptive fusion mechanism, weighted by the LVLM's prediction entropy, allows for targeted correction, applying stronger visual grounding when the LVLM is more uncertain.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.