Reproduce “DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent” on OpenResearch

About this paper

Tongzhou Wu, Yuhao Wang, Xinyu Ma, Xiuqiang He, Shuaiqiang Wang, Dawei Yin, Xiangyu Zhao

This work introduces DeepResearch-9K, a dataset of 9,000 multi-level questions, and DeepResearch-R1, an open-source training framework, to facilitate the development and evaluation of AI agents for complex web-based research. Experiments demonstrated that smaller, specialized Llama-3.2-3B agents achieved 22.50% accuracy on a challenging subset, surpassing a larger DeepSeek V3 baseline (20.18%), confirming the dataset's difficulty and the framework's utility.

Key insights

• DeepResearch-9K effectively captures real-world research complexity; its L3 tasks require a mean of 20.23 search tool calls and closely match the difficulty of the BrowseComp-Plus benchmark.
• The effectiveness of training paradigms varies across models, with some architectures (e.g., Qwen-2.5-3B) strongly benefiting from Supervised Fine-Tuning (SFT) before Reinforcement Learning (RL).
• Smaller, specialized agents like Llama-3.2-3B can achieve higher performance than larger general-purpose models (e.g., DeepSeek V3) when trained with targeted reinforcement learning and aligned reward signals.

Summary by alphaXiv.