Reproduce “Flickr30K-CFQ: A Compact and Fragmented Query Dataset for Text-image Retrieval” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haoyu Liu, Yaoxian Song, Xuwu Wang, Zhu Xiangru, Zhixu Li, Wei Song, Tiefeng Li

A new dataset, Flickr30K-CFQ, is presented to better benchmark text-image retrieval systems against real-world compact and fragmented queries, revealing significant performance reductions for current models. Accompanying this, an LLM-based query enhancement method dynamically expands concise queries, leading to consistent improvements in retrieval performance across various query granularities.

Key insights

  • Existing multi-modal pre-trained models show a substantial performance degradation (e.g., Multi-recall@10 dropping from ~78% to ~26.62%) when evaluated on Flickr30K-CFQ's compact and fragmented queries, underscoring the dataset's challenge and realism.
  • Large Language Models can effectively augment semantically deficient compact queries, providing richer contextual information for cross-modal alignment without relying on explicit knowledge bases.
  • A multi-turn voting mechanism and batch-based query processing enhance retrieval robustness and accuracy by considering multiple interpretations and expansions of the initial query, mitigating the inherent randomness of LLM generation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.