Reproduce “VLind-Bench: Measuring Language Priors in Large Vision-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Kang-il Lee, Minbeom Kim, Seunghyun Yoon, Minsung Kim, Dongryeol Lee, Hyukhun Koh, Kyomin Jung

VLind-Bench is a new benchmark for measuring language priors in Large Vision-Language Models (LVLMs), revealing that GPT-4o exhibits reduced reliance on them and that Reinforcement Learning from Human Feedback for Vision (RLHF-V) mitigates this tendency. The benchmark employs a multi-stage evaluation pipeline to disentangle the language prior phenomenon from other factors like commonsense knowledge and visual perception.

Key insights

  • VLind-Bench functions as a diagnostic tool, allowing researchers to pinpoint whether LVLM failures stem from a lack of commonsense, poor visual perception, general commonsense bias, or a specific reliance on language priors.
  • The benchmark demonstrates that language priors are a pervasive and challenging issue across almost all current Large Vision-Language Models.
  • The study validates Reinforcement Learning from Human Feedback for Vision (RLHF-V) as an effective strategy for reducing language prior reliance, encouraging models to prioritize visual information.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.