Reproduce “Multi-Object Hallucination in Vision-Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David F. Fouhey, Joyce Chai
University of Michigan researchers introduce ROPE, an automated protocol to quantify multi-object hallucination in Vision Language Models (LVLMs) by using visual prompts and diverse object distributions. Their findings indicate that LVLMs exhibit significantly higher hallucination rates when identifying multiple objects concurrently, especially in heterogeneous scenes, often resorting to language biases and shortcuts.
Key insights
- LVLMs exhibit significantly higher hallucination rates when identifying multiple objects concurrently compared to single objects.
- Models frequently exploit language biases and shortcuts, such as repeating previously inferred classes in homogeneous contexts, instead of performing true grounded instance recognition.
- Hallucination patterns persist in both seen and unseen images and are exacerbated by increased object heterogeneity, lower object salience, and a low visual modality contribution from the model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.