Reproduce “Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuan Lu, Kangle Li, Haohang Huang, Rui Meng, Wenjun Zeng, Xiaoyu Shen
The paper introduces MCMR, a new large-scale benchmark for multi-conditional multimodal retrieval that requires fine-grained attribute reasoning and cross-modal evidence. Experiments using MCMR reveal that current first-stage retrievers struggle with complex, multi-constraint queries, but MLLM-based rerankers significantly improve early-rank accuracy by explicitly verifying conditions.
Key insights
- Current first-stage multimodal retrievers often prioritize holistic semantic similarity and struggle to reliably perform fine-grained, compositional, and constraint-aware reasoning across heterogeneous modalities.
- Models exhibit significant modality asymmetries, generally relying more heavily on visual cues for initial ranking, with textual metadata primarily acting as a complementary signal that enhances overall performance.
- MLLM-based pointwise rerankers demonstrate superior ability in fine-grained relevance estimation and explicit query-candidate consistency verification, effectively bridging the semantic gap left by first-stage retrievers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.