Reproduce “Finding Distributed Object-Centric Properties in Self-Supervised Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Samyak Rawlekar, Amitabh Swain, Yujun Cai, Yiwei Wang, Ming-Hsuan Yang, Narendra Ahuja
Researchers at the University of Illinois Urbana-Champaign and collaborators analyze self-supervised Vision Transformers, finding that object-centric information is encoded across all attention components (query, key, value) and distributed hierarchically throughout the network, not just in the final layer. They introduce Object-DINO, a training-free algorithm that leverages these findings to significantly improve unsupervised object discovery, for instance, by +12.4 CorLoc on VOC 2012 for DINO-V2, and reduce object hallucination in Multimodal Large Language Models.
Key insights
- Object-centric properties are robustly encoded in similarity maps derived from all three self-attention components (query, key, and value), offering more comprehensive signals than individual components.
- Object-centric information is a distributed property, emerging hierarchically across intermediate layers of the ViT, not solely confined to the final layer.
- Not all heads in the final layer are object-centric; some introduce noise, while valuable object-centric heads consistently appear in earlier layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.