Reproduce “Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts” on OpenResearch

About this paper

Aditya Sharma, Michael Saxon, William Yang Wang

Researchers from the University of California, Santa Barbara developed LOCOVQA, a dynamic benchmark for evaluating vision-language models' ability to perform extractive reasoning over long visual contexts with distractors. Their experiments reveal that VLMs consistently exhibit a rapid, logarithmic decay in performance as the visual context length increases, struggling significantly with visual distractors.

Key insights

• Current state-of-the-art VLMs fundamentally struggle with extractive reasoning over long visual contexts containing distractors, indicating a significant 'losing visual needles in image haystacks' problem.
• The consistent logarithmic decay in performance and observed positional biases suggest that existing VLM training objectives are insufficient for developing robust selective attention and filtering capabilities across numerous visual inputs.
• The research highlights a critical gap in VLM architectures and training, demonstrating a deficiency in their ability to understand and process multimodal information effectively in noisy, extended visual environments.

Summary by alphaXiv.