Reproduce “Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Natan Bagrov, Eugene Khvedchenia, Borys Tymchenko, Shay Aharon, Lior Kadoch, Tomer Keren, Ofri Masad, Yonatan Geifman and 4 more
NVIDIA Corporation researchers developed Efficient Video Sampling (EVS), a method for pruning temporally redundant tokens in videos to accelerate Vision-Language Model (VLM) inference. EVS reduces LLM Time-To-First-Token by up to 4x and linearly decreases KV-cache memory, enabling processing of longer video sequences with minimal accuracy degradation when combined with stochastic uptraining.
Key insights
- Real-world videos contain high temporal redundancy, allowing for substantial token reduction at the patch level without significant loss of critical information.
- Preserving original positional identities for retained tokens is crucial for maintaining spatio-temporal understanding by the VLM's language backbone, especially after targeted uptraining.
- Stochastic uptraining effectively generalizes the VLM's ability to handle a wide range of pruning rates, enabling flexible latency-accuracy trade-offs at inference time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.