Reproduce “$\text{PKS}^4$:Parallel Kinematic Selective State Space Scanners for Efficient Video Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lingjie Zeng, Hailun Zhang, Xiwen Wang, Qijun Zhao
PKS4, a framework for efficient video understanding, decouples spatial and temporal modeling by integrating a 2D image-pretrained Vision Transformer with a Kinematic Selective State Space Scanner. This approach incorporates motion-aware kinematic priors to enhance temporal state transitions, achieving state-of-the-art accuracy on the SSV2 benchmark while reducing total training computation by approximately 10x compared to existing video State Space Models.
Key insights
- Decoupling spatial and temporal modeling, leveraging image-pretrained backbones for spatial semantics and linear-complexity SSMs for temporal dynamics, provides an efficient architecture for video understanding.
- Integrating explicit motion-aware kinematic priors significantly enhances the effectiveness of temporal state transitions in selective State Space Models.
- Preserving the 2D spatial structure through parallel patch-wise temporal scanning allows for efficient transfer of rich spatial knowledge from image pre-training, reducing overall computational costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.