Reproduce “Adding simple structure at inference improves Vision-Language Compositionality” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Imanol Miranda, Ander Salaberria, Eneko Agirre, Gorka Azkune

An inference-time approach enhances the Vision-Language Compositionality of dual-encoder models like CLIP and SigLIP 2 by decomposing images into crops and text into segments, then matching them. This method consistently improves retrieval performance across datasets, achieving gains such as over 22 absolute points on BIVLC and up to 90 points on controlled compositional benchmarks without additional training.

Key insights

  • Pretrained dual-encoder Vision-Language Models contain latent compositional representations that are not fully leveraged by standard global embedding comparisons.
  • Enhancing compositional reasoning can be achieved through inference-time modifications, challenging the reliance on extensive retraining.
  • Decomposing images into smaller crops is crucial for improving compositional understanding, as confirmed by ablation studies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.