Reproduce “Decouple and Cache: KV Cache Construction for Streaming Video Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhanzhong Pang, Dibyadip Chatterjee, Fadime Sener, Angela Yao

DSCache addresses streaming video understanding by maintaining a bounded cumulative cache for history and rebuilding a cache from recent frames when a query arrives. It stores entries without positional encoding and assigns positions during use. On StreamingBench, LLaVA-OneVision-7B reaches 79.12% versus 76.92% for uniform caching under a 32-frame budget, but adds latency and is not uniformly better across subtasks.

Key insights

  • Uniform KV cache construction in streaming contexts can lead to a 'cumulative effect' that degrades the informativeness of newly encoded recent inputs by retaining residual, potentially outdated, information.
  • Position-agnostic encoding provides a robust and formally proven method to handle position extrapolation and discontinuity in unbounded streams for RoPE-based LLMs by dynamically assigning position IDs.
  • Decoupling the storage and processing of fine-grained recent input features from the update of long-term cumulative past context is crucial for maintaining high-fidelity recent information in streaming video understanding.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.