Reproduce “Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning” on OpenResearch

About this paper

Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, Jiawei Han

Search-R1 trains a language model to integrate reasoning and search during question answering. In reinforcement-learning rollouts, it generates queries, receives three retrieved passages, and may search again before answering, using final-answer correctness as reward. On seven datasets, Qwen2.5-7B-Base reaches 0.431 average exact match versus RAG’s 0.304; the displayed averages imply about 42% relative improvement, while reported percentages conflict.

Key insights

• Complex multi-turn search and reasoning behaviors can be effectively learned by Large Language Models using Reinforcement Learning with a simple, outcome-based reward function.
• Applying loss masking to retrieved tokens is crucial for stable RL optimization when integrating non-differentiable search engine calls, preventing unintended optimization of external content.
• LLMs trained with this framework learn to autonomously generate valid search queries, integrate retrieved information into their reasoning, and even perform self-verification steps.

Summary by alphaXiv.