Reproduce “Leveraging Registers in Vision Transformers for Robust Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Srikar Yellapragada, Kowshik Thopalli, Vivek Narayanaswamy, Wesam Sakla, Yang Liu, Yamen Mubarka, Dimitris Samaras, Jayaraman J. Thiagarajan
This paper demonstrates that repurposing register tokens, initially designed to stabilize Vision Transformer training, enhances their out-of-distribution generalization and anomaly rejection capabilities. The method achieves 2-4% improvements in accuracy on OOD benchmarks like ImageNet-A and reduces false positive rates by 2-3% across various ViT architectures.
Key insights
- Register tokens, originally for training stability, capture global image information that is complementary to the standard CLS token.
- This complementary information from registers is particularly valuable for improving Vision Transformer performance in out-of-distribution scenarios.
- An existing architectural component can be effectively repurposed to address challenges beyond its original design intent, enhancing model robustness without significant architectural changes or retraining.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.