Reproduce “Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory” on OpenResearch

About this paper

Vatsal Agarwal, Saksham Suri, Matthew Gwilliam, Pulkit Kumar, Abhinav Shrivastava

MemStream, developed by researchers at the University of Maryland and TikTok, enhances multimodal large language models (MLLMs) for streaming video understanding by employing adaptive key selection for KV-cache encoding and a training-free retrieval Mixture-of-Experts. This method improves MLLM performance on video question answering, achieving up to an 8.5% increase in accuracy on LVBench and an 8.0% increase on CG-Bench compared to prior state-of-the-art approaches.

Key insights

• Simply increasing the number of tokens per frame in existing KV-cache-based streaming video systems degrades performance due to temporal bias, feature redundancy, and less selective attention.
• The internal attention mechanism of MLLMs is an unreliable sole source for retrieving relevant video frames, showing high variance in recall across different layers.
• Combining diverse retrieval signals from both internal MLLM representations and external specialized vision models, through rank-based fusion, significantly improves retrieval robustness and overall performance.

Summary by alphaXiv.