Reproduce “ChatSearch: a Dataset and a Generative Retrieval Model for General Conversational Image Retrieval” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zijia Zhao, Longteng Guo, Tongtian Yue, Erdong Hu, Shuai Shao, Zehuan Yuan, Hua Huang, Jing Liu
Researchers from CASIA and Bytedance introduce ChatSearch, a comprehensive dataset for general conversational image retrieval, and ChatSearcher, a generative retrieval model. The model achieves superior performance on the new dataset by interpreting multi-round, multimodal dialogues and delivers factual images as direct outputs, along with strong generalization across other multimodal tasks.
Key insights
- Leveraging multi-round, multimodal conversational context significantly improves image retrieval performance by enabling the model to discern implicit user intentions.
- A generative approach, capable of producing both text and retrieved images as outputs, effectively unifies and optimizes multimodal dialogue comprehension and information delivery.
- Diverse instruction tuning data, including visual conversation (LLaVA-150k) and AIGC-based examples (InstructPix2Pix), is crucial for enhancing the model's ability to follow complex multimodal instructions and improve retrieval.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.