Reproduce “Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models” on OpenResearch

About this paper

Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu and 1 more

Researchers from Rutgers University and Microsoft Research developed MMNeedle, a multimodal benchmark to evaluate the long-context capabilities of Multimodal Large Language Models (MLLMs) using a needle-in-a-haystack task. The benchmark revealed that while GPT-4o exhibits strong performance in simpler long-context scenarios, all evaluated MLLMs, including leading proprietary models, face significant degradation in accuracy with increasing context length and complexity, often showing hallucination in negative samples.

Key insights

• Increasing visual context length, whether through more images or greater stitching density, consistently leads to a substantial decline in MLLM performance, indicating limitations in long-range attention and processing.
• A significant performance gap exists between proprietary API-based MLLMs (e.g., GPT-4o) and open-source models, especially in complex multi-image and multi-needle scenarios.
• MLLMs, even high-performing ones, are prone to hallucination in negative samples, incorrectly 'finding' a needle that does not exist, which poses challenges for reliable real-world deployment.

Summary by alphaXiv.