Reproduce “Skip-Convolutions for Efficient Video Processing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amirhossein Habibian, Davide Abati, Taco S. Cohen, Babak Ehteshami Bejnordi
Skip-Convolutions optimize deep neural network inference for video by selectively computing convolutions on residual frames (differences between frames or feature maps) using learned or norm-based gating functions. This approach achieves a 3-4x reduction in computational cost for object detection and a 2x reduction for human pose estimation, typically maintaining or slightly improving accuracy.
Key insights
- Repurposing convolution to explicitly process residual information across frames effectively exploits temporal redundancy inherent in video data.
- Conditional computation applied to residuals yields significantly higher efficiency gains compared to image-based conditional computation, highlighting the rich sparsity of temporal changes.
- Learned Gumbel gates can adaptively identify task-relevant areas for computation, leading to superior efficiency and sometimes improved accuracy, potentially acting as a regularization mechanism.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.