Reproduce “Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ioanna Ntinou, Alexandros Xenos, Yassine Ouali, Adrian Bulat, Georgios Tzimiropoulos

LexiCLIP introduces a vision-free approach to multimodal search by transforming images into rich textual descriptions using Large Vision-Language Models, achieving state-of-the-art retrieval performance with a much smaller text-only model. This framework, developed by researchers at Queen Mary University of London, Samsung AI Centre, and Technical University of Iasi, improves compositional understanding and addresses privacy concerns inherent in traditional vision-language models.

Key insights

  • High-performance multimodal image-text retrieval is achievable without a dedicated vision encoder by representing all visual content as rich textual descriptions.
  • Operating solely within the linguistic domain effectively bridges the modality gap, leading to superior compositional understanding and reduced reliance on surface-level textual matching.
  • The vision-free paradigm significantly enhances privacy by avoiding direct image data processing and drastically reduces computational training costs, requiring only light finetuning on text samples.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.