Reproduce “Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, Ge Yu
UniVL-DR proposes a unified model for dense retrieval across text and image modalities, embedding queries and documents into a shared space. This approach achieves over 7% higher MRR@10 compared to previous multi-modal retrieval models and outperforms oracle-routed pipeline systems, demonstrating the effectiveness of a single end-to-end framework.
Key insights
- Unifying multi-modal information retrieval into a single embedding space, rather than separate pipelines, is more effective and can outperform oracle-guided systems.
- Mitigating modality bias through balanced hard negative sampling is critical for training robust universal dense retrievers.
- Image verbalization, particularly generating query-like text, effectively enhances textual understanding of images and bridges the cross-modality gap.
- Training on multi-modal data for a universal representation space can yield performance benefits even for single-modality and cross-modality retrieval tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.