Reproduce “ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Le Tu Ngoc Minh, Jinyeong Lim, Dongsu Han
ProtoKV introduces a streaming video understanding (SVU) method that combines a fixed-size near-exact memory window with a fixed-capacity far-summary prototype bank to address delayed queries. This approach significantly improves robustness to long query delays, achieving up to 20.4 percentage points higher accuracy over sliding-window methods on SVU benchmarks while maintaining a constant memory footprint and low query-time latency.
Key insights
- Summarizing distant history into a fixed-capacity prototype bank, rather than retaining individual token instances, is crucial for improving robustness to delayed queries under strict memory constraints.
- Capturing intra-prototype diversity through residual statistics and applying mass-aware attention biasing for pseudo-tokens effectively leverages aggregated summary information for improved query performance.
- A two-tier memory architecture, combining an exact near-window for immediate events and a continuously updated, query-agnostic far summary, provides an effective balance between fidelity for recent information and durability for distant evidence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.