Reproduce “Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reduction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Patrick Glandorf, Thomas Norrenbrock, Bodo Rosenhahn
The Video Patch Pruning (VPP) framework from Leibniz University Hannover introduces a method for efficient Vision Transformer operation in Video Instance Segmentation by enabling early-stage patch reduction through a temporal mapping module. It achieves up to 60% patch reduction, reducing FLOPs by 32% and increasing backbone FPS by 51% (e.g., from 17.14 to 25.9 at 55% PKR), while maintaining high Average Precision with minimal degradation (e.g., 0.6% AP drop at 55% PKR on Youtube-VIS 2021).
Key insights
- Early Vision Transformer layers exhibit low "Foreground Selectivity," meaning they struggle to distinguish foreground from background, rendering conventional image-based early pruning ineffective.
- Leveraging temporal prior knowledge, specifically deep features from previous frames, is crucial for accurately guiding early-stage patch pruning in video, enabling higher sparsity without significant performance loss.
- A fully differentiable temporal mapping module can effectively perform motion compensation and sparsely activate background patches, ensuring detection of new instances while maintaining foreground focus.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.