Reproduce “UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, Yu Qiao
UniFormer presents a unified Transformer architecture for video understanding that efficiently integrates local 3D convolutional feature aggregation in shallow layers with global spatiotemporal self-attention in deeper layers. This adaptive design achieves state-of-the-art accuracy on Kinetics-400 and Something-Something benchmarks while reducing computational costs by up to 10x compared to leading approaches.
Key insights
- An adaptive strategy that combines local feature aggregation in shallow network layers with global dependency modeling in deeper layers achieves a superior computation-accuracy trade-off for video.
- Interpreting local self-attention as a channel-separated spatiotemporal convolution enables efficient handling of local redundancy in early processing stages.
- Joint spatiotemporal self-attention, rather than separate spatial and temporal attention, yields more discriminative video representations when applied to abstract features in deep layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.