Reproduce “Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach” on OpenResearch

About this paper

Saehyung Lee, Sangwon Yu, Junsung Park, Jihun Yi, Sungroh Yoon

Researchers at Seoul National University developed PlugIR, a plug-and-play framework for interactive text-to-image retrieval that leverages Large Language Models to adapt dialogue context for pre-trained Vision-Language Models without fine-tuning. The system achieved superior performance on multiple datasets, consistently outperforming both zero-shot and fine-tuned baselines, and introduced a new evaluation metric called Best log Rank Integral (BRI) that aligns with human judgment.

Key insights

• Traditional zero-shot Vision-Language Models (VLMs) struggle to effectively utilize dialogue-form inputs, often treating conversational context as noise.
• Grounding Large Language Model (LLM) generated questions in the visual content of retrieval candidates significantly improves query relevance and efficiency.
• A new evaluation metric, Best log Rank Integral (BRI), correlates highly with human judgment for interactive retrieval systems, better capturing user satisfaction and efficiency.

Summary by alphaXiv.