Reproduce “Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models” on OpenResearch

About this paper

Zhuowan Li, Cihang Xie, Benjamin Van Durme, Alan Yuille

Researchers from Johns Hopkins University and UC Santa Cruz quantitatively analyzed visual representations from vision-and-language (VL) and vision-only (V) models, finding that VL models exhibit stronger semantic understanding while V models demonstrate superior spatial localization capabilities, with VL models achieving over 51% accuracy on object name prediction compared to V models' less than 50%.

Key insights

• A clear trade-off exists: multimodal learning with language imbues visual representations with superior semantic understanding, while vision-only self-supervised learning excels at learning fine-grained spatial localization.
• Language supervision primarily enhances the ability of visual representations to predict abstract attributes and object names, rather than improving raw spatial accuracy.
• Task-specific finetuning on downstream tasks tends to decrease the general, fine-grained information encoded during pretraining, suggesting a specialization effect rather than broad improvement.

Summary by alphaXiv.