Reproduce “Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

Research reveals Multimodal Large Language Models (MLLMs) struggle to accurately answer visual questions requiring perception of small visual details in zero-shot settings, exhibiting a significant accuracy drop as object size decreases. Inference-time visual cropping methods are proposed to mitigate this limitation, improving zero-shot VQA performance on detail-focused datasets and suggesting MLLMs often correctly identify relevant regions but fail to perceive fine-grained information.

Key insights

  • MLLMs exhibit a substantial decline in zero-shot VQA accuracy (e.g., 45.91% for BLIP-2 FlanT5_XL) as the relative size of the question's visual subject decreases, confirming a significant perception bottleneck.
  • The limitation is causally linked to visual detail size, as 'zooming in' via cropping significantly improves performance, especially for small objects.
  • MLLMs often 'know where to look' by identifying salient regions through internal attention/gradient mechanisms even when providing incorrect answers, suggesting a failure in perceiving fine details within the identified region rather than in initial localization or question understanding.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.