Reproduce “Memory-enhanced Retrieval Augmentation for Long Video Understanding” on OpenResearch

About this paper

Huaying Yuan, Zheng Liu, Minghao Qin, Hongjin Qian, Yan Shu, Zhicheng Dou, Ji-Rong Wen, Nicu Sebe

MemVid, a memory-enhanced retrieval augmentation framework, processes long videos by combining a holistic memory with a reasoning module to infer implicit information needs, outperforming existing long-context Vision-Language Models and RAG methods. It achieves state-of-the-art results among 7B-parameter models across multiple benchmarks while significantly reducing computational costs.

Key insights

• Emulating human cognitive processes of holistic memorization, reasoning, and selective attention is highly effective for overcoming the limitations of brute-force context expansion and explicit query reliance in long video understanding.
• Designing a flexible, reasoning-oriented memory module that generates structured retrieval clues allows for efficient and targeted extraction of critical information from vast video content, bridging the gap between implicit needs and explicit retrieval.
• Optimizing an intermediate module (the memorizer) without direct ground truth labels can be achieved through a progressive curriculum learning framework that combines supervised foundation with reinforcement learning based on downstream generation feedback, aligning intermediate outputs with end-to-end performance.

Summary by alphaXiv.