Reproduce “Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback” on OpenResearch

About this paper

Zheng Wang, Bingzheng Gan, Wei Shi

A new Multimodal Query Suggestion (MMQS) task is introduced, where an image serves as input for generating textual search suggestions. Researchers at Huawei Singapore Research Center developed RL4Sugg, a multi-agent reinforcement learning from human feedback framework, which achieved up to an 18% improvement in DCG over existing models and led to a 0.46% increase in user click behaviors upon real-world deployment.

Key insights

• Explicitly separating the optimization of intentionality and diversity into two distinct, cooperatively trained reinforcement learning agents effectively addresses the complex, multi-objective MMQS problem.
• Leveraging large language models (LLMs) as policy networks, conditioned on visual inputs, enables the generation of semantically rich and contextually relevant textual suggestions.
• Integrating a confidence-based automated labeling strategy with human validation significantly reduces the burden of data annotation for multimodal tasks.

Summary by alphaXiv.