Reproduce “MileBench: Benchmarking MLLMs in Long Context” on OpenResearch

About this paper

Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, Benyou Wang

MILEBENCH is a pioneering benchmark introduced by researchers from The Chinese University of Hong Kong, Shenzhen, to evaluate Multimodal Large Language Models (MLLMs) in long-context scenarios. It reveals significant performance gaps between closed-source (e.g., GPT-4o achieving 99.4% in diagnostic tasks) and open-source models, especially concerning multi-image processing and optical character recognition within extended multimodal inputs.

Key insights

• Closed-source MLLMs, especially GPT-4o, demonstrated a substantial performance lead over open-source models across both diagnostic and realistic evaluations, indicating superior long-context adaptability and information retrieval.
• Most MLLMs exhibited a significant performance decline as the number of images per sample increased, suggesting a general lack of generalization capacity for multi-image test data due to insufficient training on such scenarios.
• The "Lost in the Middle" phenomenon, previously identified in unimodal language models, was also observed in multimodal contexts for less capable MLLMs like Qwen-VL-Chat, though advanced models such as GPT-4V were able to avoid it.

Summary by alphaXiv.