Reproduce “Seeing Through Words: Controlling Visual Retrieval Quality with Language Models” on OpenResearch

About this paper

Jianglin Lu, Simon Jenni, Kushal Kafle, Jing Shi, Handong Zhao, Yun Fu

Researchers from Adobe Research and Northeastern University introduce Quality-Conditioned Query Completion (QCQC), a framework leveraging generative language models to enrich short user queries with quality-aware descriptions for text-to-image retrieval. The method consistently enhances both the aesthetic and relevance scores of retrieved images in alignment with specified quality levels, outperforming post-retrieval filtering approaches.

Key insights

• Generative LLMs can be effectively fine-tuned to serve as quality-conditioned query completion functions, creating semantically rich and quality-aware instructions for VLMs.
• Finetuning the LLM for specific quality control is crucial, as general-purpose or unfinetuned LLMs often degrade retrieval performance by generating irrelevant terms.
• Integrating quality control into the query generation process (pre-retrieval) is more effective than applying it as a post-retrieval filtering or reranking step, as it guides the VLM from the outset.

Summary by alphaXiv.