Reproduce “Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yulin Wang, Rui Huang, Shiji Song, Zeyi Huang, Gao Huang
The Dynamic Vision Transformer (DVT) framework enables Vision Transformers to adaptively adjust the number of image tokens processed based on input image complexity, leading to substantial computational efficiency gains. This approach reduces theoretical FLOPs by up to 3.6x on ImageNet and achieves 1.4-1.7x faster inference for larger models while maintaining or improving accuracy.
Key insights
- Adaptive allocation of tokens per image, based on content complexity, significantly improves computational efficiency in Vision Transformers without sacrificing accuracy.
- Reusing deep features and learned attention relationships across cascaded Transformers is critical for effectively minimizing redundant computation and enhancing performance in a dynamic token setting.
- Dynamically adjusting the number of tokens is a more effective strategy for efficiency in Vision Transformers than simply varying network depth (adaptive depth).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.