Reproduce “Revisiting the Role of Language Priors in Vision-Language Models” on OpenResearch

About this paper

Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, Deva Ramanan

Researchers from Carnegie Mellon University and Meta introduced Visual Generative Pre-Training Score (VisualGPTScore) and an alpha-debiasing scheme, effectively repurposing generative Vision-Language Models for discriminative tasks like image-text retrieval. This approach achieves state-of-the-art results on several benchmarks, rigorously exposes pervasive linguistic biases in existing evaluations, and provides a training-free solution for more robust visio-linguistic reasoning.

Key insights

• Generative VLMs, when evaluated with VisualGPTScore, inherently possess strong capabilities for image-text alignment and compositional reasoning, often surpassing complex discriminative VLM solutions.
• Linguistic biases are pervasive in many existing VLM benchmarks, leading to misleadingly high performance from models that exploit unnatural negative examples rather than true visio-linguistic understanding.
• Controlling for language priors via the alpha-debiasing framework is crucial for accurate VLM evaluation, especially on benchmarks with realistic negative captions, and can significantly improve true visio-linguistic reasoning.

Summary by alphaXiv.