Reproduce “QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Design” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Benjamin Schneider, Dongfu Jiang, Chao Du, Tianyu Pang, Wenhu Chen
QuickVideo, developed by researchers at the University of Waterloo and SeaAI Lab, is a system-algorithm co-design framework that substantially accelerates real-time long video understanding with VideoLLMs. It achieves over a 3x speedup for a 30-minute video compared to a sequential baseline by optimizing video decoding and LLM prefilling, while maintaining high frame sampling rates and reducing memory usage.
Key insights
- A system-algorithm co-design is essential to address the combined bottlenecks of video decoding and LLM prefilling for long videos.
- Parallelizing video decoding by leveraging keyframe intervals significantly accelerates the process, especially for dense sampling rates common in VideoLLMs.
- Group-based prefilling and selective KV cache pruning based on 'Key Norms (small)' effectively reduce memory footprint and computation for extremely long sequences while largely preserving accuracy.
- Overlapping CPU-intensive video decoding with GPU-intensive LLM prefilling is critical to achieving substantial end-to-end latency reductions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.