Reproduce “TransFG: A Transformer Architecture for Fine-grained Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang
TransFG adapts Vision Transformer for fine-grained image classification by combining attention across layers to select patch tokens, then processing them with the class token in a final Transformer layer. It also uses contrastive loss. On CUB-200-2011, it achieves 91.7% Top-1 accuracy, versus 90.3% for ViT-B/16. It exceeds ViT on five datasets, but its 94.8% on Cars trails PMG and API-Net.
Key insights
- Vision Transformers, when augmented with specific modifications like an overlapping patch strategy and an attention-based part selection mechanism, are effective for fine-grained visual classification tasks.
- Integrating attention weights across multiple transformer layers offers a robust and intrinsic method for automatically identifying discriminative image regions, bypassing the need for complex external localization modules.
- Combining an explicit part selection mechanism with a contrastive loss significantly enhances the model's ability to learn highly discriminative features for classes with minimal visual differences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.