Reproduce “Bridge the Modality and Capability Gaps in Vision-Language Model Selection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chao Yi, Yu-Hang He, De-Chuan Zhan, Han-Jia Ye
Researchers from Nanjing University developed SWAB (VLM Selection With gAp Bridging), a framework that accurately selects Vision-Language Models for zero-shot image classification tasks using only textual descriptions. It addresses the "Modality Gap" between image and text embeddings and the "Capability Gap" in VLM performance variability, achieving a 14.8% improvement in R5+τ score over the prior state-of-the-art.
Key insights
- The existence of both a Modality Gap and a Capability Gap fundamentally limits the accuracy of language-only VLM selection, and explicitly addressing them is crucial.
- Optimal transport effectively quantifies semantic relevance between classes, enabling robust transfer of fine-grained VLM statistics like modality gap vectors and performance rankings.
- Class-level modality gap vectors are more consistent and informative than dataset-level vectors, improving the effectiveness of modality gap bridging.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.