Reproduce “Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hang Wu, Sherin Mary Mathews, Yujun Cai, Ming-Hsuan Yang, Yiwei Wang
Researchers at the University of California, Merced, and collaborators developed SAVEMem, a training-free framework for online streaming video question answering that integrates query-agnostic semantic memory generation with adaptive, query-aware retrieval. The method achieved a +10.41 point gain on the OVO-Bench for the Qwen2.5-VL-7B backbone and reduced peak GPU memory usage by 48% at 128 frames, enhancing efficiency for continuous video streams.
Key insights
- Injecting a lightweight, query-agnostic semantic prior (via a pseudo-question bank) during memory compression can effectively preserve semantically salient visual tokens over long video streams.
- Decoupling memory generation from retrieval, and then adaptively tuning the retrieval scope based on the query's temporal relevance using a recency gate, enhances efficiency and accuracy for diverse query types without fine-tuning.
- A hierarchical three-tier memory combined with selective forgetting and distinct semantic-aware pruning/selection strategies for each tier enables robust memory management under strict memory constraints, achieving sub-linear token growth.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.