Reproduce “Probing Visual Language Priors in VLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tiange Luo, Ang Cao, Gunhee Lee, Justin Johnson, Honglak Lee
Researchers from the University of Michigan and LGAI Research introduced ViLP, a benchmark designed to diagnose visual language prior over-reliance in Vision-Language Models, and Image-DPO, a self-improving framework to enhance visual reasoning. ViLP revealed a substantial performance gap between VLMs and humans, while Image-DPO improved VLM visual grounding, general VQA performance, and reduced object hallucination.
Key insights
- VLMs, including advanced models like GPT-4o, exhibit a significant performance gap compared to humans when presented with visual evidence that contradicts strong language priors.
- Systematically corrupting images or introducing 'distractor facts' reveals VLMs' sensitivity to visual quality and their susceptibility to being misled by textual biases.
- Optimizing VLM preferences based on image quality (Image-DPO) rather than answer quality effectively compels models to develop stronger visual grounding, improving reasoning and mitigating hallucinations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.