Reproduce “Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models” on OpenResearch

About this paper

Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Renrui Zhang, Yi Wang, Yu Qiao, Hongsheng Li

The Retrieving-to-Answer (R2A) framework addresses Video Question Answering by retrieving relevant textual information from generic text corpora using pre-trained vision-language models and then prompting frozen Large Language Models. This approach achieves state-of-the-art zero-shot performance across multiple benchmarks with significantly reduced computational requirements compared to prior methods.

Key insights

• The quality of the external text corpus significantly impacts performance, with human-annotated captions outperforming larger web-scraped data.
• Performance improves with an increasing number of retrieved captions per video, showing benefits even from a single caption and diminishing returns after 20-50 captions.
• The R2A framework effectively leverages the inherent capabilities of more powerful language models, showing larger accuracy gains when using stronger LLMs.

Summary by alphaXiv.