Reproduce “DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learning” on OpenResearch

About this paper

Wenxuan Shi, Haochen Tan, Chuqiao Kuang, Xiaoguang Li, Xiaozhe Ren, Chen Zhang, Hanting Chen, Yasheng Wang and 2 more

Huawei Language Model Lab introduces DeepDiver, an RL-driven framework, and WebPuzzle, a new open-web benchmark, enabling large language models to dynamically adjust their web search intensity for complex information-seeking. DeepDiver-enhanced 7B-parameter models achieved 37.6-38.1% accuracy on WebPuzzle, matching or exceeding the performance of significantly larger models like the 671B-parameter DeepSeek-R1 (37.1%) and outperforming existing Wiki-based methods.

Key insights

• Reinforcement learning can foster emergent adaptive behaviors like Search Intensity Scaling (SIS) in LLMs, enabling them to proactively use external tools to compensate for internal knowledge limitations, even after direct search incentives fade.
• A carefully designed curriculum, transitioning from a loose to a stricter reward function, is crucial for stabilizing early RL training and effectively overcoming performance plateaus in complex, real-world information-seeking tasks.
• Training on an open-web dataset like WebPuzzle promotes strong generalization, allowing models to perform well not only on complex real-world queries but also on out-of-distribution wiki-based problems and open-ended generation tasks.

Summary by alphaXiv.