Reproduce “Improving fine-grained understanding in image-text pre-training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer and 3 more
SPARC addresses a mismatch in image–text pretraining by grouping image patches with caption tokens and training local matches alongside whole-image and whole-caption alignment. It compares each token with patches from its paired image, using sparse weights to form token-specific visual summaries. On zero-shot PASCAL VOC segmentation, it reaches 27.36 mIoU, above CLIP’s 23.02, showing improved localization in this evaluation.
Key insights
- Fine-grained understanding can be efficiently integrated into vision-language models without sacrificing global context or incurring prohibitive computational costs.
- Sparsifying the patch-token similarity matrix and grouping visual embeddings by language tokens effectively captures one-to-many visual-textual relationships.
- Combining global and fine-grained contrastive objectives leads to improved performance across diverse downstream tasks, including localization and generative applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.