Reproduce “Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tingyu Song, Yanzhao Zhang, Mingxin Li, Zhuoning Guo, Dingkun Long, Pengjun Xie, Siyue Zhang, Yilun Zhao and 1 more
EDIR, a new benchmark for Composed Image Retrieval (CIR), is introduced, featuring a fine-grained hierarchical taxonomy of image modifications and a programmatic data synthesis pipeline. Evaluation reveals current multimodal models struggle with compositional reasoning and fine-grained details, while existing CIR benchmarks exhibit significant modality and evaluation biases.
Key insights
- Current multimodal embedding models, including MLLM-based architectures, exhibit considerable limitations in compositional reasoning, handling negation, and sensitivity to fine-grained visual details.
- Existing CIR benchmarks are plagued by 'fine-grained evaluation bias' due to imbalanced categorical coverage and a 'modality bias' that enables models to exploit text-only shortcuts.
- In-domain training on EDIR reveals that some challenges are solvable with targeted data, but others, particularly those requiring complex compositional reasoning, indicate intrinsic architectural weaknesses not easily resolved by data alone.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.