Reproduce “ALLVB: All-in-One Long Video Understanding Benchmark” on OpenResearch

About this paper

Xichen Tan, Yuanjing Luo, Yunfan Ye, Fang Liu, Zhiping Cai

Researchers from National University of Defense Technology and Hunan University introduced ALLVB, an "All-in-One Long Video Understanding Benchmark," designed to evaluate Multi-modal Large Language Models (MLLMs) on extended video content. This benchmark features 1,376 movies, averaging nearly two hours in duration, with 252,420 automatically generated Q&As, revealing that even state-of-the-art MLLMs like Claude 3.5 Sonnet (75.2% accuracy) and GPT-4o (68.0% accuracy) exhibit significant performance gaps in complex long-form video understanding tasks, particularly those requiring fine-grained detail retrieval.

Key insights

• Automated Q&A generation using large language models like GPT-4o, coupled with multi-stage quality control, provides a scalable and reliable method for constructing large-scale, complex benchmarks.
• Current state-of-the-art MLLMs, even commercial models, demonstrate significant limitations in deep comprehension and reasoning over ultra-long video contexts, especially for fine-grained visual and precise temporal tasks.
• MLLM performance on long videos is sensitive to both increasing video length and limited input frame count, indicating a need for improved long-context processing and temporal modeling strategies.

Summary by alphaXiv.