Reproduce “LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenyu Ning, Guangda Liu, Qihao Jin, Chengwei Li, Wenchao Ding, Minyi Guo, Jieru Zhao
LiveVLM is a training-free and query-agnostic framework for online video understanding, introducing Vision Sink Bucketing for efficient KV cache compression and Position-agnostic KV Retrieval. It achieves leading accuracy among online models, improving by 4.25% on StreamingBench compared to its foundation model, while reducing peak GPU memory by 3.02 times and speeding up Time To First Token by 1.73 times against baselines.
Key insights
- Vision-to-vision attention patterns exhibit 'sink vision tokens' that consistently attract high attention, making their selective retention crucial for maintaining attention outputs more effectively than text-to-vision scores.
- Discontinuous positional embeddings, caused by token discards in compressed caches, severely impair the similarity of key tensors within pages, necessitating a mechanism to disregard positional information during retrieval.
- Balancing high attention scores with broad contextual coverage through a bucketing strategy is essential to prevent retaining only locally relevant but globally redundant tokens in the compressed cache.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.