Reproduce “TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ji Lin, Chuang Gan, Kuan Wang, Song Han
A Temporal Shift Module (TSM) developed by Lin et al. from MIT allows 2D Convolutional Neural Networks to capture temporal relationships in videos without adding computational overhead. This module achieves accuracy comparable to computationally expensive 3D CNNs, demonstrating up to 10x fewer FLOPs, and improved performance by up to 31.3% on temporal-heavy datasets like Something-Something V2.
Key insights
- Temporal convolution can be effectively achieved by simple data shifting along the temporal dimension, followed by standard 2D convolutions.
- Shifting only a subset of channels (e.g., 1/4) balances temporal information flow with spatial feature preservation and minimizes latency.
- Integrating temporal shifting within residual blocks allows for robust temporal modeling while maintaining compatibility with existing 2D CNN architectures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.