Reproduce “Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Baiyang Song, Yuli Lin, Qiong Wu, Tao Chen, Jun Peng, Xiao Chen, Yiyi Zhou, Rongrong Ji
A training-free method, CausalMem, enables efficient streaming video understanding for MLLMs by implementing a dynamic, fixed-budget visual memory bank. This approach improves LLaVA-OneVision accuracy by 3.1% on OVO-Bench and 3.2% on StreamingBench, and allows Qwen2.5-VL to exceed GPT-4o's performance on these benchmarks by over 3%, while significantly reducing computational overhead.
Key insights
- An online semantic basis can causally and effectively identify redundant visual tokens in a streaming video, maximizing information within a limited memory space.
- Maximizing retained information in a fixed-budget memory bank for streaming video understanding requires balancing semantic novelty (redundancy) with temporal recency.
- A training-free, causal memory management system can be a plug-and-play enhancement for existing MLLMs, significantly improving their streaming video understanding capabilities without additional training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.