Reproduce “VLM-Eval: A General Evaluation on Video Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuailin Li, Yuang Zhang, Yucheng Zhao, Qiuyue Wang, Fan Jia, Yingfei Liu, Tiancai Wang
VLM-Eval introduces a comprehensive, human-validated evaluation framework for Video Large Language Models, combining GPT-based and retrieval-based metrics across diverse tasks. It also proposes Video-LLaVA, a simple linear connector model, which achieves state-of-the-art performance on various benchmarks and demonstrates strong capabilities in autonomous driving scenarios with limited fine-tuning.
Key insights
- GPT-3.5 can reliably and automatically assess open-ended Video LLM responses, showing strong correlation with human judgments and significantly reducing the evaluation burden.
- Simple architectural designs, such as a linear connector in Video-LLaVA, can achieve state-of-the-art performance, challenging assumptions about the necessity of complex video-to-text adapters.
- Video LLMs exhibit strong potential for practical, real-world applications like autonomous driving, demonstrating capabilities in open-set recognition, spatial-temporal reasoning, and safety suggestions with minimal fine-tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.