Reproduce “A Sanity Check on Composed Image Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yikun Liu, Jiangchao Yao, Weidi Xie, Yanfeng Wang
Researchers from Shanghai Jiao Tong University developed a comprehensive evaluation suite for Composed Image Retrieval (CIR) models, introducing the Fully-Informed Semantically-Diverse (FISD) benchmark to address ambiguous queries and an automated multi-round framework to assess interactive performance, revealing that multi-round interactions consistently enhance model efficacy.
Key insights
- Existing CIR models show notable deficiencies in handling complex semantic modifications, such as negation and cardinality, even on the precisely controlled FISD benchmark.
- Multi-round interactions consistently lead to substantial performance gains for CIR models across various benchmarks, indicating significant untapped potential in iterative refinement.
- Foundation models can effectively simulate human user feedback in multi-round evaluation settings, offering a scalable and realistic method for assessing interactive retrieval systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.