Reproduce “Rethinking Vision Transformer for Large-Scale Fine-Grained Image Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xin Jiang, Hao Tang, Yonghua Pan, Zechao Li
An Efficient and Effective Transformer-based framework (EET) is introduced for large-scale fine-grained image retrieval, which incorporates content-based token pruning and a discriminative transfer strategy to reduce the computational overhead of Vision Transformers. This approach significantly lowers ViT-Small inference latency by 42.7% on the NABirds dataset while improving retrieval mAP by 3-7% compared to other methods.
Key insights
- Prioritizing tokens based on the L2 norm of their content combined with class attention effectively identifies and preserves discriminative information crucial for fine-grained tasks.
- A teacher-student paradigm, incorporating both knowledge distillation and region-based guidance, can effectively mitigate accuracy degradation caused by significant token pruning in Vision Transformers.
- Progressive token pruning across transformer blocks, mimicking human visual attention from global to local focus, allows for capturing both overall context and fine-grained details efficiently.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.