Reproduce “Efficient Training of Visual Transformers with Small Datasets” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, Marco De Nadai
Researchers at the University of Trento and Fondazione Bruno Kessler investigate the performance of advanced Visual Transformers on small datasets and introduce Dense Relative Localization (L_drloc), a self-supervised auxiliary task. L_drloc consistently boosts the accuracy of VTs across various datasets and tasks, particularly in data-limited scenarios, with negligible computational overhead.
Key insights
- Second-generation VTs show substantial performance variability when trained from scratch on medium-small datasets, contrasting with their similar performance on ImageNet-1K.
- The proposed `L_drloc` effectively acts as a regularizer, consistently improving VT accuracy and generalization, especially under data or computational scarcity.
- Architectures like CvT-13 demonstrate greater inherent robustness on smaller datasets compared to Swin-T and T2T-ViT-14, sometimes matching or exceeding ResNet-50 performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.