Reproduce “AIM: Adapting Image Models for Efficient Video Action Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, Mu Li
This paper introduces AIM, an efficient method for adapting pre-trained image transformer models to video action recognition. By freezing the image backbone and integrating lightweight adapters that reuse existing self-attention for temporal modeling, AIM achieves competitive performance on major benchmarks with substantially fewer trainable parameters and reduced computational overhead.
Key insights
- Pre-trained image self-attention layers can be efficiently reused for temporal modeling in video tasks by re-orienting their input sequence, requiring minimal additional parameters.
- Introducing lightweight, trainable 'Adapters' within a frozen image transformer backbone is sufficient to imbue it with robust spatiotemporal reasoning capabilities for video action recognition.
- Parameter-efficient transfer learning can yield superior or competitive performance compared to full finetuning, especially in low-data regimes, while drastically reducing computational costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.