Reproduce “Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifei Yuan, Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke, Wai Lam
Researchers developed Marto, a multimodal generative model that enhances conversational search by enabling systems to ask clarifying questions augmented with relevant images. This approach improved document retrieval performance by up to 90% and led to more detailed user responses, supported by the introduction of the large-scale Melon dataset.
Key insights
- Incorporating images into clarifying questions significantly enhances document retrieval effectiveness and user response quality.
- Intelligent classification of which questions benefit from images, and the selection of the single most relevant image, are crucial for optimal performance.
- Generative retrieval models offer superior effectiveness and substantial efficiency gains compared to discriminative models in multimodal conversational search.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.