Reproduce “Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siting Li, Xiang Gao, Simon Shaolei Du
Researchers from the University of Washington and Tsinghua University introduced promptable image embeddings, where an MLLM conditions image representations on textual prompts, to address the limitation of current text-to-image (T2I) retrievers in handling fine-grained, attribute-focused queries. This approach achieved substantial performance gains on a new benchmark, COCO-FACET, increasing average Recall@5 from 58.9% to 75.5% while offering practical acceleration strategies.
Key insights
- Existing state-of-the-art T2I retrievers exhibit significant performance drops on attribute-focused queries, often prioritizing dominant but irrelevant features.
- Prompting MLLM-based universal embedders effectively guides the model to focus on attribute-relevant information, enhancing fine-grained retrieval without architectural changes.
- Practical acceleration methods can make promptable embeddings viable for real-world T2I retrieval systems by reducing the computational overhead of dynamic prompting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.