Reproduce “ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning” on OpenResearch

About this paper

Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z. Pan and 5 more

ReSearch trains language models with reinforcement learning to choose and refine searches during multi-hop question answering, using retrieved results to guide subsequent reasoning without labeled reasoning traces. On HotpotQA, ReSearch-Qwen-32B-Instruct scores 67.70% on GPT-4o-mini’s semantic-correctness judgment, versus 58.80% for Iter-RetGen. Results cover four benchmarks in Wikipedia-based retrieval, not arbitrary real-world use.

Key insights

• Reinforcement learning, driven by straightforward outcome-based rewards, can effectively teach LLMs to autonomously perform interactive, multi-step reasoning by leveraging external search tools.
• Structuring search operations within the LLM's thought process using explicit tags fosters a robust and dynamic interaction with external knowledge sources.
• Advanced reasoning capabilities, such as self-elicited reflection and self-correction of search queries, can naturally emerge from the reinforcement learning process without explicit design or training for these behaviors.

Summary by alphaXiv.